After forcing a Power off and Exit and rebooting, access to the Flint VM machine via IE yielded the report. I do not understand why the VM Machine hangs up. Is this a Flint problem?

There is limited feedback while parsing and processing tests, a UI issue we hope to address soon, so it is likely that it was actually doing those task when you perceived it was hung. When you reboot it, it will come back up, with partial results, and thus not attempt to parse/check again, and therefor was responding immediately.

Once you upload the file, there will then be a parse phase, and after that you will get a screen titles "Tell us a little about your firewal..." and then once you tell it to run the checks it will perform the longer analysis phase. With our larger rule sets this can take 5 minutes or more.

I have restarted the same reoprt from the beginning. The parser phase ran and completed almost immediately. The "Tell us .." screen came up and I clicked the "Run checks" " button without makig any changes. The lower left corner reads "Wiiting for httP://:4567/preprocess ...". It has been running for 20 minutes. It will be left to chug along during the night.(I'd color it hung but the morning will tell.)

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

Which verison of IE is this?

Unless your rule set is like 20k lines or something, it should not be taking that long.

I'm thinking there is a UI/Ajax/Post issue, because once you submit the
"Run Checks" it should very quickly redirect you to the /run_tests URL. It seems like your browser is still waiting for preprocess to return.

How about hitting the "stop" button and clicking on the "Overview" tab?

The IE version is 8.0.6001.18904.

The firewall config file has 2630 lines.

When I left last night Flint was still in the preprocessor mode. When I came in this morning the screen was on an error screen indicating a broken link. Clearing and reentering the URL took me back to the Overview screen and what looked like a finished report.

I then tried to rerun the same report in order to try what you said above. The IE session is again in a wait state; this time waiting for "overview".

I also do not understand the Rule Syntax errors. The report says "No Syntax Errors for Rules" but there are pages of lines that say "Parse Error".

Now what?

Richard

I've gone back to look at the configuration file to see if the parse error made any sense. Each of the lines in error is a remark. For example, one line reads "access-list outside_access_in remark New smtp server Steve 7-09".

Should they be flagged as Parse Errors?

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

> The IE version is 8.0.6001.18904.

Ok, I'll fire this up under IE8 and see if I can reproduce this problem.

> I then tried to rerun the same report in order to try what you said > above. The IE session is again in a wait state; this time waiting for > "overview".

Can you try this with a different browser please to confirm that it is
an IE 8 issue? We use Safari, Chrome and Mozilla for the dev here. I
don't have any other reports of this issue from VM users.

> I also do not understand the Rule Syntax errors. The report says "No > Syntax Errors for Rules" but there are pages of lines that say "Parse > Error".

Our parser is not complete for all PIX commands, only those that effect
the firewall behavior model we build. We ony make parse errors on those
commands, like access-list, etc... as a syntax error.

-- Craig Brozefsky
craig@matasano.com
Senior Software Developer

Tried to upgrade via the VM. When rake install command was entered a series of messages such as "ruby-debug (0.10.3) is cached, but not installed" were displayed. The last message was "Try running bundle install" was displayed. Tried that. Received message "... could not find RubyGem bundler (Gem::LoadError)

Is there a command missing or is it best to wait for the 1.06 VM? I don't want to be a pain in the rear but as stated before I'm not knowledgeable with Linux or Ruby.

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

> Is there a command missing or is it best to wait for the 1.06 VM? I > don't want to be a pain in the rear but as stated before I'm not > knowledgeable with Linux or Ruby.

You can redownload the FlintVM-current.zip and that should be
the 1.0.6 VM.

The OVF version of the VM is still uploading tho.

-- Craig Brozefsky
craig@matasano.com
Senior Software Developer

After downloading version 1.0.6 I tried running the same firewall check as before. The program is still hanging up. After waiting 20 minutes the VM was powered down and rebooted. Entering via IE and logging into Flint it was still hung. Pressing stop and clicking the overview tab needed to be done twice before the report appeared.

The syntax error on the Remarks have vanished. It is interesting to note that the number of "External-to-Internal Service Checks" errors have changed from 1 passed and 50 failed to 9 failed.

What can I do to assist is trouble shooting the "hang" problem?

Can other browsers such as Firefox or Google Chrome be used? (I wouldn't expect the browser to be the cause of the "Hang" issue.)

Richard

Try using a different browser please. I was able to reproduce this with IE8 when using a large rule set, but I do not get the same hang when using Mozilla and the same large ruleset.

To help diagnose this, you can bring up the log under "Settings" You can cut and paste the first couple hundred lines into a message here in Lighthouse.

20 minutes is too long for a 1200 line file. My 4k line files don't take that long.

I tried using Mozilla Firefox but after 30 minutes Flint was still hung. I powered off the VM and rebooted. I tried to access the log file via the settings button but there is nothing there.

Am I doing something wrong in this process? You may have to give me a list of step-by-step instructions or we can do it by telephone.

Note my configuration file is 121KB in size - 2629 lines.

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

How much memory does the host running the VM have?

The FlintVM wants 512M to run, so if there is not 512M of real ram available to the VM then it will start swapping and that could lead to this hang. Since Redis writes out it's contents to disk, that would explain why you are getting at least partial results after a reboot.

So your log file is totally empty? That is bizzare.

Tired using Google Chrome - same results. Crashed VM machine with CTRL+Alt+DEL. Reentered Flint via Chrome. Got the following on screen. Not sure if this is what you want.

./login.rb in nil

    redirect "/overview " unless User[session[:user]].is_admin?

/usr/lib/ruby/1.8/webrick/httpserver.rb in service

  si.service(req, res)

/usr/lib/ruby/1.8/webrick/httpserver.rb in run

      server.service(req, res)

/usr/lib/ruby/1.8/webrick/server.rb in start_thread

      block ? block.call(sock) : run(sock)

/usr/lib/ruby/1.8/webrick/server.rb in start

  Thread.start{

/usr/lib/ruby/1.8/webrick/server.rb in start_thread

  Thread.start{

/usr/lib/ruby/1.8/webrick/server.rb in start

              th = start_thread(sock, &block)

/usr/lib/ruby/1.8/webrick/server.rb in each

          svrs[0].each{|svr|

/usr/lib/ruby/1.8/webrick/server.rb in start

          svrs[0].each{|svr|

/usr/lib/ruby/1.8/webrick/server.rb in start

  yield

/usr/lib/ruby/1.8/webrick/server.rb in start

  server_type.start{

app.rb in nil
get "/" do
GET
No GET data.
POST
No POST data.
COOKIES
Variable Value
rack.session
"BAh7CDoQY3VycmVudF9zaGEiLWVlZTAzYTAwYmRlNDM2OGM1NDQwYjk2MmFh\nYjkxMTZhNDhiNzc4Y2Q6DmxvZ2luX3VyaSIGLzoJdXNlciIGMQ==\n--bf4f73267775492e81b9896af04a16b3fd541971" Rack ENV
Variable Value
GATEWAY_INTERFACE
CGI/1.1
HTTP_ACCEPT
application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5

ISO-8859-1,utf-8;q=0.7,*;q=0.3

gzip,deflate,sdch

en-US,en;q=0.8
HTTP_CONNECTION
keep-alive
HTTP_COOKIE
rack.session=BAh7CDoQY3VycmVudF9zaGEiLWVlZTAzYTAwYmRlNDM2OGM1NDQwYjk2MmFh%0AYjkxMTZhNDhiNzc4Y2Q6DmxvZ2luX3VyaSIGLzoJdXNlciIGMQ%3D%3D%0A--bf4f73267775492e81b9896af04a16b3fd541971
HTTP_HOST
151.141.73.80:4567
HTTP_REFERER
http://151.141.73.80:4567/upload

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1045 Safari/532.5
HTTP_VERSION
HTTP/1.1
PATH_INFO
/admin QUERY_STRING
REMOTE_ADDR
151.141.73.60
REMOTE_HOST
etsu904089.etsu.edu
REQUEST_METHOD
GET
REQUEST_PATH
/ REQUEST_URI
http://151.141.73.80:4567/admin
SCRIPT_NAME
SERVER_NAME
151.141.73.80
SERVER_PORT
4567
SERVER_PROTOCOL
HTTP/1.1
SERVER_SOFTWARE
WEBrick/1.3.1 (Ruby/1.8.6/2007-09-24)
rack.errors

<IO:0xb7c85f5c>

rack.input

<StringIO:0xb6d56720>

rack.multiprocess
false
rack.multithread
true
rack.request.cookie_hash
{"rack.session"=>"BAh7CDoQY3VycmVudF9zaGEiLWVlZTAzYTAwYmRlNDM2OGM1NDQwYjk2MmFh\nYjkxMTZhNDhiNzc4Y2Q6DmxvZ2luX3VyaSIGLzoJdXNlciIGMQ==\n--bf4f73267775492e81b9896af04a16b3fd541971"} rack.request.cookie_string
rack.session=BAh7CDoQY3VycmVudF9zaGEiLWVlZTAzYTAwYmRlNDM2OGM1NDQwYjk2MmFh%0AYjkxMTZhNDhiNzc4Y2Q6DmxvZ2luX3VyaSIGLzoJdXNlciIGMQ%3D%3D%0A--bf4f73267775492e81b9896af04a16b3fd541971
rack.request.query_hash
{} rack.request.query_string
rack.run_once
false
rack.session
{:login_uri=>"/", :current_sha=>"eee03a00bde4368c5440b962aab9116a48b778cd", :user=>"1"} rack.session.options
{:path=>"/", :expire_after=>nil, :key=>"rack.session", :domain=>nil, :secret=>"0504e91e4c9d13f9f2b0fa13b66c35400ef3267f"} rack.url_scheme
http
rack.version
[1, 1] sinatra.error

<Errno::ECONNREFUSED: Connection refused - Unable to connect to Redis on 127.0.0.1:6379>

You're seeing this error because you use you have enabled the show_exceptions option.

My machine is running Vista with 2GB of memory. When running FlintVM the task manager shows the CPU running at 54% using 1.29GB. FlintVM is still hung.

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

What does crashing it with CTRL+ALT+DEL mean? I read that as killing
the VM guess process, as opposed to doing a soft powerdown (which lets
the Guest OS sync it's disks and go thru it's normal shutdown).

It looks like we can rule out the VM mem starvation, you said with
FlintVM running you were using 1.29G.

Something that caught my eye was the 50 failed external-to-internal
checks. That indicates that your firewall is relatively permissive,
which means that it could be generating a large amount of ServicePath
and ProtocolMap objects, which record which protocols can go thru the
firewall to/from sets of hosts.

Can you login to the VM (user flint, password flint) and do the
following and send me the output:

ls -l ~/flint

That will tell me the size of the dump.rdb file, which is Redis' on disk
store.

The error backtrace indicated that Redis did not start up. Redis is
kind of like Flint's database server, except that all the data is kept
in memory at once, and it writes it out to disk from time to time. It's
possible that it would fail to start up if the data was larger than the
memory available (just under 512M). If that dump.rdb is over 400M,
maybe you could try giving the VM even more memory, your VMWare software
should let you up it, and it seems your host has the headroom.

Also, the log that would help me is in ~/flint/app/logs/flint.log on the
VM. You can login with the user flint, password flint. You can
navigate to that in your browser, by going to settings, and then there
is a link on that page to view the logs. Alternatively you can get to
it by navigating to http://&lt;vmaddress&gt;/logview

Unfortuantely, I don't think that is goin to help us much. The log will
tell us wether it's in parsing, analyzing or checking phase, but not
much beyond that. It may contain info on a Redis crash if that is the
cause.

My next development task is to add alot more logging to the analysis
code in Flint so that users get more feedback than just a spinner in
their browser.

Thanx for the time you've spent on this so far. It is greatly
appreciated.

-- Craig Brozefsky
craig@matasano.com
Senior Software Developer

I don't get much of anything when I try to get the documentation requested. Attached is a WORD doc with a some screen shots.

Let me know if there is anything else I can do.

Richard

I've downloaded version 1.0.7 and am running into the same "hang" problem as before. This time the IE screen status in the lower left corner indicated Done but did not change the Overview page. I had to power off the VM and reboot. When I reentered the Flint screen after signing on the screen showed the report overview. Attached you will find a PDF copy of the report. My best guess is that it is probably correct but still no clue for the hang up.

Richard

Does your 1.0.7 have a logs directory in ~/flint/app?

or at the url:

http://<vmaddress>/logview

If so, can you send those along.

I get nothing when I go to the URL. There is no logs directory in ~/flint/app. There is a file named app.log in the ~/flint directory

Richard

Sorry, there is currently a bug with logging. Flint logs are apparently being stored in /var/log under the file flint.log. We'll get this fixed as soon as possible. In the meantime, could you look under this directory and send us the log file?

Thanks

Joel

Attached is a screen capture of the flint.log. I don't think it shows a whole lot.

Richard

It appears to me that Flint is running the External-to_Internal Service checks. The report seems to indicate a different number of errors each time the system is interrupted and rebooted. It has been restarted and the plan is to let it run for a few hours, only occasionally checking the browser for screen changes.

(Craig noted above "That indicates that your firewall is relatively permissive, which means that it could be generating a large amount of ServicePath and ProtocolMap objects, which record which protocols can go thru the
firewall to/from sets of hosts." This firewall is on a University network. There are 75 object-group network statements and 75 object-group service statements in the configuration file.)

Richard

The report appears to have completed normally after running approximately 2.5 hours.

Is there any other information you need regarding the configuration I am checking?

Richard

Lighthouse <no-reply@lighthouseapp.com> writes:

Wow, that is a really long time, more than an order of magnitude than
we're seeing for other large configurations.

Let me think about this a bit more, and I'll write a script that will
gather some statistics about the data Flint generated from that config.
This ought to help us figure out what is slowing it down so much.

How long does it take if you don't run the external->internal checks?

Also, how long does it take if you disable ALL the checks?

Thanx again for your patience and perseverence.

Running the reports with External-to-Internal checks disabled took 3 minutes. The report with All checks disabled also ran in 3 minutes.

Attached are the full report and the report with no external checks.

I see you have alot of interfaces on the device. Could you tell us all the interfaces you have in the external, dmz and internal realms?

Could you give us a timing for running the external->internal checks, but when you only have one outside interface, and one inside interface, and the rest marked as "none" or "unknown"?

There are a 150 or so protocol checks that are done for each path from the outside to the inside. So if you have 2 external interfaces and 3 internal interfaces, that would be 6 * 150 checks.

I made a script today that will count the number of objects generated from running your config. To get this script into your flint vm, you will need to update it via the update feature under administration. In order for updates to take effect, you must shutdown the vm and power it back up.

Once this is done, run your config, navigate to flint/script and execute the dbstats script. Send us the output generated from this script.

This should give us an idea about how much data Flint is dealing with from your configuration.

I have downloaded and extracted Flint 1.0.8 from the website. Do I still need to perform the update feature step above? When I list the contents of the flint/script directory there is nothing there named dbstats.

Richard

You do need to run the update in order to get a copy of dbstats. This file has been pushed onto the releases branch, but has not been packed with the linked files on our site. Let me know if you have any problems updating or obtaining this script.

Joel

I've updated the software and started the firewall report. After running for more than an hour with no screen change the VM was powered off and rebooted. When I accessed the report the External-to-internal errors were a different figure from the last time indicating the report was not complete. Attached is a screen shot of the dbstats.

I'll rerun the report tomorrow until complete and get the stats again.

Do the statistics reset automatically when a new report is generated?

Richard

Now the Flint VM does not boot up correctly. Attached is a screen shot of the console. What gives?

Richard

Now the Flint VM does not boot up correctly. Attached is a screen shot of the console. What gives?

Richard

Sorry about the VM failing after the update. It appears this failure is caused by a hard shut-down of the guest OS.

To (temporarily) fix this, run the manual fsck and type 'y' for each suggested fix, and control+D when finished. Looks like we need to package vm-tools in our next release to facilitate soft-shutdown of the guest OS.

I'm glad you were able to run dbstats at least once. It gave us a much better idea of what your configuration is doing to Flint.

Also, thanks for helping us debug the update feature! We apologize for the inconvenience.

Joel

More trouble!

Flint VM was rebooted and the same report started. After running about 2.5 hours it was noted that the VM was no longer up and IE was still waiting for a response from it. Restarting the VM apparently returned it immediately to the same status when it went down. The attached error report was then displayed on IE.

I really don't mind helping as I'd like to see a decent tool but this is getting a bit cumbersome.

Richard

Going back to your comment on May 4th.

You mentioned that you downloaded and extracted the 1.0.8 release.

Do you mean that you downloaded the tarball source distribution and extracted that into your existing Flint VM?

Flint 1.0.8 will resume the analysis phase when you make a request for the overview page, even after startup. Your browser has a cookie which tells it which firewall you are currently looking at, and then when you request that firewall, it will run the checks if they were not run already. This is why it starts back up when you reboot.

I just completed some work that will change that behavior, and will also let you watch the progress of your checks, while Flint does all the analysis in the background.

By downloading the 1.0.8release I mean that the entire VM zip file was downloaded from the ttp://runplaybook.com/flint website.

On May 4, after downloading the VM it was updated per Joel instructions in order to implement the DBSTATS program. There was then problems rebooting.(See his entry on May 4, 9:27 PM). It has not been touched since - either software or report testing.

Next step?

Richard

Have you tried running fsck in the vm console when these problems arise? See my May 4th post for more details on this process.
If you have gone through this process and are still unable to run Flint, let me know.

Otherwise, next time you process the report, run it with only one external and one internal interface selected. All other interfaces should be set to 'other'. Let me know how long it takes to process your report with these settings.

Joel

I guess I've done enough to where the VM boots "normally", going through its recovery process.

A report with "DMZ1" checked as the external interface and "Inside" checked as the internal interface and all checks selected ran in about 1 minute.

Currently a report is running with "inside" checked as the internal interface and "outside" checked as the external interface. Only "External_to_internal_service_checks" are selected. It has been running for about 20 minutes. I will be leaving the office for the day in about 20 minutes. If it has not finished I will let it continue and check it in the morning. Of course I will not be able to tell you how long it ran. I will run DBSTATS when I check the VM machine in the morning.

Richard

Attached are a few screen shots from the report tests.

Richard