Neohaxor.org

This year at Black Hat USA 2010 and Defcon 18 Marcin Wielgoszewski and I did a talk called Constricting the Web: Offensive Python for Web Hackers (video). The basic premise of our talk is that web architectures and technology are getting far more complicated and it is not sufficient just to run a vulnerability scanner on an application and call it done. Individuals tasked with testing these architectures are going to need to write their own tools and tests at some point. If you aren’t taking security beyond your vulnerability scanner then you aren’t performing the proper due care and due diligence required to protect your assets.

More information on tools and projects to come. I just don’t feel too much like writing today

Write Your Own

It’s inevitable at some point you are going to have to write your own tools and tests. Modern web architectures no longer consist of a page with a simple backend web server. Complex items such as RIA technologies, APIs, aggregators, and custom protocols are now thrown in to the mix. Vendors continue to lag behind the technology curve which puts commercial tools at a disadvantage. All of these items need to be tested in order for organizations to have any type of success in their testing efforts.

This is where you take the code in to your own hands. It really isn’t as difficult as it may sound. I know quite a few people that shy away at the thought of writing their own code. I am not quite sure why there is so much apprehension about writing your own code. Maybe people are having C flashbacks from college or something.

Modern languages have quite a bit of the work already done for you. Python, which is my weapon of choice, has a vast amount of modules that allow interfacing with many different protocols. In the end all you need to do send and record your tests. Python is also a rapid development language that is easy to read and write, making it great for security people who don’t want to spend all day writing code.

Black Hat Wrap-up

We are participating in the Black Hat Wrap-up webcast. We are going to summarize our talk and highlight a few items. This is happening Thursday, August 19th at 4pm Eastern. Information on the webcast can be found here: https://www.blackhat.com/html/webcast/webcast-2010_BHUSAwrapup.html

Tools Released

We also released a couple of tools. Most notable is Marcin’s Burp API. It allows you to interface with Burp logs and turn them in to objects. This would allow you to do anything from replaying tests, creating your own macros, and even creating your own vulnerability scanner. Burp API information can be found here: http://mwielgoszewski.github.com/burpee/

I also released a stand alone encoder that I wrote a while back. The main reason was I just like having a stand alone encoder when I am doing assessments. It allows you to encode and decode values as well as wrap values with different characters. You can get more information on DharmaEncoder here: http://code.google.com/p/dharmaencoder/

Finally, I created a Python web fuzzing module called pywebfuzz. The pywebfuzz module allows you to have values available for testing from the fuzzdb project as well as some convenience functions for range generation and making requests. The module is in it’s early phases but still usable. I have a bunch of miscellaneous things I need to do to it before it is where I would like it to be. More information on pywebfuzz can be found here: http://code.google.com/p/pywebfuzz/

If you run in to bugs please let us know so we can get them fixed. If there are features you would like to be added please let us know that as well.

Conference Materials

The conference materials are posted on the Hexagon Security site. You can download them here: http://hexsec.com/docs

I haven’t really written much about Python lately. I have a feeling that is about to change Python is great because it is powerful and allows you to do things very quickly. I figured I would write a short post to show how to remove duplicates by just using the set type. This is probably the quickest and easiest way of removing duplicates in Python.

I don’t think I need to get in to how useful it can be to easily remove duplicates. I have used this many times in the past for doing everything from removing duplicate values from a list of SQL Injection checks to just determining how many unique occurrences I have for a given test.

set

Python has a type called a set. A set is basically an unordered collection of unique values. You can create a set by specifying a new empty set and adding values to it or by converting another type. The set conversion can be done over any iterable object.

Create a new empty set called myset:

myset = set()

You can add values to your set by using add or update:

myset.add("hello")
myset.update("world")

Convert another type to a set called newset:

newset = set(another_type)

Sets in Python are nice for a couple of reasons. The first being they only keep unique values. This means that any type you convert to a set or anything you add to a set is unique. It will discard duplicate values. Secondly, you can test for membership in the set. Testing for membership will give you a True / False response based on whether a value exists in the set.

Here are some examples

Converting a list to a set.

Even strings are iterable objects in Python. String conversion to set.

The following shows True / False values for membership tests from the previous string conversion.

Practical Example

Let’s say you wanted to write a small program that took a file, removed the duplicates, and created a new file with only unique values. The file that contains the duplicates has one value per line, which means there is a newline at the end of each item. You want to maintain the newline in the new unique file that you are writing to as well. You will see the newlines in the following specified by “\n”.

The following is an example:

#!/usr/bin/env python
 
import sys
 
if len(sys.argv) < 3:
    print "Usage: remove_dups.py original_file.txt unique_file.txt"
    sys.exit(1)
 
file1 = open(sys.argv[1])
file2 = open(sys.argv[2], "w")
 
unique = set(file1.read().split("\n"))
 
file2.write("".join([line + "\n" for line in unique]))
file2.close()

I will explain a bit of what’s happening here. Let’s say we have copied this in to a file called remove_dups.py. This program takes two arguments, your original file and the name of the file you want to create without the duplicates. If it doesn’t have the two arguments the program exits.

Next both files are opened, with the second file opened for writing. The first file is read in splitting on newlines. The unique variable now contains the unique values. We then write to the second file every line concatenating a newline on the end. This makes the second file contain the unique values one per line.

You now now how to remove duplicates in Python using the set type. Knowing is half the battle

Update

I wrote this post very quickly and didn’t explain about my use of read() vs using readlines(). Marcin pointed out yesterday that it wasn’t clear. I wanted to show how you could use read() and split on newline characters. My hope was that you would see how you could split on any character when reading a file like commas, semicolons, asterisks, or anything really.

In the code example above, if you wanted to read in a file per line instead of splitting on the “\n” character you could just use readlines() instead.