Page 28 of 28

Nmap Tutorial

Jan 13, 2006

Port scanning is a method of finding listening services (open ports) on a computer. In laymen’s terms, port scanning is akin to checking if doors and windows are unlocked on a house. It is useful for locating vulnerable computers on a network for security auditing purposes as well as attacking them. TCP diagram

To understand port scanning, you first must know how the TCP protocol works. TCP is called a connection oriented protocol, as it initiates a connection between two machines when they want to transfer data. This is compared to connectionless protocols, like UDP, which don’t initiate connections or gurantee delivery of data. To begin a connection, the client sends a packet to the server with the SYN flag set which tells the server it wants to initiate a connection and where to start numbering packets. The server responds with a packet that has the SYN and ACK flags set, acknowleding the connection. The client then completes the setup of the connection by responding with a packet having the ACK flag set. After this point, the client and server can communicate as a stream, sending all the data they wish. When they have completed sending all of their data, the connection is terminated when one of them sends a FIN/ACK packet and the other responds with an ACK.

Now that we’ve covered the basics of TCP, we can move on to Nmap itself. Nmap is a featureful port scanner supporting all of the popular scanning mechanisms including TCP connect(), SYN, FIN, Xmas and null scans. In addition, Nmap supports ping sweeping, operating system detection, decoy and zombie hosts.

The TCP connect() scan is the simplest and Nmap’s default. All it does is check each port on the target machine by initiating a connection using the operating system’s connect() call. Being that it only utilizes standard system calls available to normal users, this type of scan does not require root or administrator priveleges. However, the connect() is easily detected because it is fairly noisy; the port scanner must fully initiate a connection with each port that it finds open.

A SYN scan, also knowsn as a half-open scan, only sends the first packet of a connection initiate, the SYN packet. It never attempts to complete the connection, rather it simply listens for the response of the target. If it replies with a SYN/ACK packet, the port is in listening mode. If it responds with a RST/ACK, the port is closed. The SYN scan is considered to be a stealth scan, as the connection is never completed and therefore never registered with any listening services. Most modern firewalls and monitoring systems can detect SYN scans, however.

FIN, Xmas tree and null scans all work in the same manner. The port scanner sends a packet to the target machine out of the blue, which doesn’t make sense out of the context of a connection. RFC 793 stipulates that closed ports should respond to this probe with a RST whilst open ports will ignore it. Unfortunately, this doesn’t work against Microsoft operating systems because they don’t comply to the standard, ignoring such packets instead.

Because of their nature, attempting to connect to numerous ports on a computer, port scans can usually be detected quite easily. Most port scanners will scan in a random order, so that their activities aren’t blatantly obvious, but the speed and manner with which they scan is usually still easily detected. Many firewalls have options to detect and slow or prevent port scans. Linux’s current kernel-based firewall, Netfilter (IPChains) has a number of rules that can do this, such as rules to limit port connections over a time period. There are also more advanced programs, like snort and Port Scan Attack Detector (PSAD) which can monitor for port scans, notify you of them and take action (blocking the scanning hosts) to prevent such scans.

Darwin Kernel Compile

Jan 13, 2006

Compiling the Darwin Kernel

I’m writing this how-to beause when I wanted to enable MAC address spoofing on my Powerbook, it was a little bit more of an adventure than I had hoped for. I had to look through a number of different tutorials, picking and choosing what I knew from my experience with Mac OSX and Linux/BSD to be correct; there are a number of conflicting directions in other tutorials. Hopefully mine makes sense the whole way through.

Prepare the Build Environment

To build a Mach kernel you need a number of things, namely, XCode, bootstrap_cmds, Libstreams, cctools, kext-tools, IOKitUser and xnu. The former is included on any new Mac (run the “Developer Tools” installer in the Applications folder) and the latter must be obtained from the Apple developers site, which requires a (free) membership login.

Assuming you have XCode installed, the first thing to do is grab the packages that you’ll need for building the kernel. They can be obtained from the abovementioned location, http://www.opensource.apple.com/darwinsource/tarballs/apsl. You’ll need the following packages:
- bootstrap_cmds
- Libstreams
- cctools
- kext-tools
- IOKitUser
- xnu Just grab those packages (get the latest versions) and toss them into a directory where we can work. I used /usr/local/src.
Untar all of those packages:
```
# cd /usr/local/src; for i in $(ls); do tar zxvf $i; done
```
Whew, that was tough. Go make a sandwich before continuing.
Let’s build what we need in bootstrap_cmds and Libstreams, because they’re easy.
Note: I’m performing all of these commands as root, which isn’t enabled be default on Mac OS X (I think). Most people will probably more comfortable with untarring/building all of these things in their own userspace (say, your home directory) and then using sudo when you need to be root. I come from Linux, where Real Men use su.
# cd bootstrap_cmds-*/relpath.tproj
# make
# make install
Now, Libstreams:
# cd ../../Libstreams-*
# make
# make install
Time to build cctools, it’s a little bit more involved.
# cd ../cctools-*
# cp /usr/include/sys/unistd.h /System/Library/Frameworks/Kernel.framework/Headers/sys/ We only care about some of what cctools has to offer, so bust open it’s Makefile and lets edit it a little. We’re going to comment out the two lines (28 and 29) that define COMMON_SUBDIRS and redefine it asCOMMON_SUBDIRS = libstuff libmacho misc.
#COMMON_SUBDIRS = libstuff as gprof misc libmacho ld libdyld \
# mkshlib otool profileServer RelNotes man cbtlibs
COMMON_SUBDIRS = libstuff libmacho misc That’s what it should look like when you’re done, at which point you’re ready to build and install it.
# make RC_OS=macos
# cp misc/seg_hack.NEW /usr/local/bin/seg_hack
# cd ld
# make RC_OS=macos kld_build
# cp static_kld/libkld.a /usr/local/lib
# ranlib /usr/local/lib/libkld.a
Compiling kextsymboltool is where I ran into trouble, well, because I grabbed the wrong IOKit version (I didn’t have the newest one), so watch out for that. If you get the right version, everything should go smoothly.
# cd ../../
# mv IOKitUser* IOKit
# mv IOKit /usr/include
# mv /usr/include/IOKit/kext.subproj /usr/include/IOKit/kext
# cp -fpr cctools*/include/mach-o /usr/include
# cp -fpr cctools*/include/mach/* /usr/include/mach
# cd kext_tools*
# gcc kextsymboltool.c -o kextsymboltool
# cp kextsymboltool /usr/local/bin
# chmod 755 /usr/local/bin/kextsymboltool

Done! Well, sort of. The build environment is all set up, so now if you ever want to build a new kernel, you can start on the next step.

Building the Kernel

The first thing to do is apply any patches that you desire. You’ll probably be adding some patches; otherwise, why would you be rebuilding the kernel? I’m using a patch to enable setting the MAC address of my network cards manually (for MAC address spoofing), one that boots my mac in verbose mode by default and a third that allows all users (not just root) to craft and send raw packets. They are available from the following locations (the direct links are for xnu version 517):
- BIOCSHDRCMPLT - MAC address patch by Jeff Nathan
- FORCE_VERBOSE - Verbose booting patch by Peter Bartoli
- RAW4ALL - Raw packet creation patch also by Peter Bartoli
  To begin with, let’s move to the kernel source directory:
  # cd ../xnu*
  Grab the three patches:
  # wget http://cerberus.sourcefire.com/%7Ejeff/archives/patches/macosx/BIOCSHDRCMPLT-10.3.3.patch
  # wget http://slagheap.net/etherspoof/517/FORCE_VERBOSE.patch
  # wget http://slagheap.net/etherspoof/517/RAW4ALL.patch
  and patch the kernel source:
  # patch -p0 -b --verbose --suffix=.orig < RAW4ALL.patch
  # patch -p0 -b --verbose --suffix=.orig < FORCE_VERBOSE.patch
  # patch -p0 -b --verbose --suffix=.orig < BIOCHSHDRCMPLT-10.3.3.patch Pay attention to the output of those patches; if the source changes significantly from the time I wrote this page, they might not apply properly. If they work fine, you can move on.
Finally, we will actually build the kernel:
# make exporthdrs
# make
The new kernel is now in BUILD/obj/RELEASE_PPC/mach_kernel under the current directory. Next, we’ll install and prepare to boot with the new kernel.
To install, just copy the new kernel to the root (/) directory. We’ll name it differently from the stock kernel, so that you have something to fall back on if your new kernel fails to boot properly.
# cp BUILD/obj/RELEASE_PPC/mach_kernel /mach_kernel.new
# chmod 644 /mach_kernel.new
# chown root.wheel /mach_kernel.new
Now configure the machine to boot from the new kernel:

# nvram boot-file=\nvram boot-device | awk -F , ’{print $1}’ | awk ’{print $2}’`,mach_kernel.new`

Note: That is supposed to work, but really, didn’t for me. I had to make a leap-of-faith and replace the stock kernel with the one that I just built.
# cp mach_kernel mach_kernel.bak
# cp mach_kernel.new mach_kernel
# nvram boot-file=""
and then reboot.

You should be all set! Upon reboot, the computer should boot with your new kernel. In the event something goes wrong, you can zap the PRAM by holding option-apple-p-r through three startup chimes. Alternatively, hold option-apple-o-f which will boot into OpenFirmware; type “reset-all” and then “mac-boot”. If neither of those fixes the problem, hold option-apple-n-v during boot to reset the NVRAM.

Sources

Filesystems

Jan 13, 2006

This is the outline to a seminar that I gave at CSH on filesystems. The impetus was simply my lack of knowledge for how filesystems work and the misconceptions (particularly about journaling) that most people have. Synopsis: use ReiserFS, but don’t believe the Gentoo kiddies who say you’ll get a massive performance boost; you wont. ReiserFS is slightly faster than EXT3, a little bit more space efficient and gets beyond the 2GB filesize limit. I run EXT2 because it’s a touche faster than all the journaling filesystems and I know that my systems aren’t going to go down uncleanly. Most people, however, aren’t as meticulous as I am so a Journaling filesystem is a better fit.

Filesystem writes
1. Non-journaling (EXT2, FAT)
  1. Update meta-data (size)
  2. Update indirect mapping blocks
  3. Write data
2. Journaling modes
  1. Journal
    1. Logs data and metadata changes
    2. Slowest, most secure
    3. Requires each write to be made twice
      1. Written to journal
      2. Then written to disk
    4. Log line only deleted once the transaction is done
  2. Ordered (default for EXT3)
    1. Only logs changes to metadata
    2. Writes data updates before metadata is changed
    3. If metadata wasn’t updated, the change never happened
  3. Writeback (default for ReiserFS and XFS)
    1. Only logs changes to metdata
    2. Uses regular write process
    3. Causes problems if metadata is written but data isn’t
EXT2 - non-journaling Linux
1. Maximum file size of 2GB
2. Supports maximum filesystem size of 4TB
3. Maximum file name size of 255 characters
4. Supports regular files, directories, devices and symlinks
5. Files created within a directory inherit attributes (permission, owner, group)
6. Allows user defined block sizes
  1. Typically 1024, 2048 or 4096 bytes
  2. Larger blocks speed up I/O; fewer requests issued, less head movement
7. Fast symlinks
  1. Target name stored in inode, not data area
  2. Max size of link is 60 character
8. Superblocks track clean/not clean state of filesystem
  1. Marked not clean when mounted read/write
  2. Marked clean on unmount or remounted read only
  3. Uses mount counter; after X mounts, ignores state and forces fsck
9. Includes support for secure deletion; random data written to deleted blocks
10. Physical structure - disk divided into number block groups (see Fig. 1)
  1. Each block group contains redundant filesystem info and data for the block (see Fig. 2, reference 3)
    1. Super block - total number of blocks in filesystem, check counter, etc.
    2. Group descriptor - pointers to block bitmap, inode bitmap, and inode table
    3. Block bitmap - tells which blocks are in use
    4. Inode bitmap - tells which inodes are in use
    5. Inode table - each file has an inode; stores file attributes
      1. Create time (ctime)
      2. Modify time (mtime)
      3. Permissions
      4. Owner
      5. Type (regular file, directory, device, symlink)
      6. Where file is stored on disk (see Fig. 3)
        
        Fifteen pointers in each inode
        
        First thirteen point to actual data blocks
        
        Fourteenth is indirect pointer - points to a block of pointers
        
        Fifteenth is doubly indirect - points to a block of pointers which point to blocks of pointers
    6. Data blocks - actual data is here
EXT3 - EXT2 with journaling
1. Built on top of EXT2; it just adds journaling
2. Supports all three modes of journaling
ReiserFS - Journaling with good small file performance
1. Maximum file size of 8TB
2. Maximum filesystem size of 16TB
3. Writeback journaling by default; supports ordered and journaled modes
4. Metadata organized in B+ trees
5. Allocates space for file size exactly, rather than in blocks
6. Tail packing
  1. Small files and less-than-block-sized tails are stored in the B+ tree
  2. Small file accesses are much faster
  3. Slightly more efficient use of storage space (~5% comp. to EXT2)
  4. Overall performance hit (it’s always repacking)
XFS
1. Maximum file size of 9 exabytes
2. Maximum filesystem size of 18 exabytes (64-bit mode)
3. Metadata organized in B+ trees
4. Dynamically allocated inodes
5. Block size 512 bytes to 64 kilobytes
6. Uses writeback journaling
7. Delayed allocation
  1. Data to be written is cached in RAM
  2. Space is reserved for data, but location is not
  3. Once enough data is collected (or an entire file) extents are found and data is written to the disk
  4. Helps improve allocation of single, contiguous regions for files
8. Guranteed rate I/O - allows apps to reserve bandwidth
9. Physically arranged into allocation groups (much like EXT2 block groups)
  1. Each group is mostly independent, managing it’s own free space
  2. Act like transparent sub-filesystems
  3. Each allocation group has two B+ trees
    1. One manages extents (ranges) of free space
    2. Another to manage inodes
UFS - BSD loves thee
1. Physical Structure
  1. Boot block
    1. Only in first cylinder group
    2. 8k, contains information for booting from this filesystem
    3. Blank if said filesystem isn’t used for booting
  2. Superblock
    1. Size of the filesystem
    2. Label (name)
    3. Size in blocks
    4. Date of last update
    5. Cylinder group size
    6. Data blocks per cylinder group
    7. Summary
  3. Cylinder group structure
    1. Copy of the superblock
    2. Cylinder group header (free/used table, etc.)
    3. Inodes
    4. Data blocks
ZFS - Hott
1. Endian-neutral - magic allows the filesystem to work on SPARC and x86
2. Fully 128-bit - More addressable bits than god can count
3. Negates the need for external volume managers
NTFS - very little known
FAT 12/16/32, VFAT - like EXT, but stupid
ISO 9660 - CD-ROM Filesystem
Comparison
1. Small files
  1. UFS and EXT2/3 have to allocate at least 1 kilobyte blocks; lost storage efficiency
  2. ReiserFS significantly faster
Tweaks (see reference 4)
1. Noatime
2. ReiserFS - notail; swap usage for efficiency
3. Tmpfs - /dev/shm
  1. Filesystem stored in RAM/swap space
  2. Like ramdisks, but hotter
4. Stack mounting - mount an already-mounted filesystem somewhere else
Not covered
1. OCFS - Oracle filesystem
  1. Hybrid filesystem/raw device access
  2. Databases do their own caching/read ahead; having an FS cache is bad
  3. http://otn.oracle.com/tech/linux/pdf/Linux-FS-Performance-Comparison.pdf

Sources

A Non-Technical Look Inside the EXT2 File System. - http://www.linuxgazette.com/issue21/gx/ext/layout.gif
Filesystems HOWTO. - http://www.tldp.org/HOWTO/Filesystems-HOWTO.html
Analysis of the EXT2fs Structure. - http://www.nondot.org/sabre/os/files/FileSystems/ext2fs/
Advanced Filesystem Implementor’s Guide. - http://www-106.ibm.com/developerworks/library/l-fs.html
ReiserFS Docs - http://p-nand-q.com/download/rfstool/reiserfsdocs.html

<li class=“ww-preformatted

Scalability in the XFS File System.

http://oss.sgi.com/projects/xfs/papers/xfsusenix/index.html

<li class=“ww-preformatted

Getting Started with XFS Filesystems.

http://oss.sgi.com/projects/xfs/papers/gettingstartedwithxfs.pdf

<li class=“ww-preformatted

Understanding Filesystem Types.

http://uw713doc.sco.com/en/FSadmin/CONTENTS.html

<li class=“ww-preformatted

NodeWorks Encyclopedia: UFS.

http://pedia.nodeworks.com/U/UF/UFS/

Wikipedia. - http://en.wikipedia.org

Computers

Jan 13, 2006

Tutorials
Information that I have written concerning computers and programming.

Darwin Kernel Compile - A how-to for compiling your own kernel under Mac OSX. Now you can snub off all of those people who say it’s not a real OS unless you run a custom kernel. The reason you probably want to build your own Mac kernel is to enable MAC address spoofing, which I cover.
Nmap - An introduction to portscanning and Nmap.
Filesystems - I gave a seminar about filesystems for CSH members. I became quite well versed in the low-level operation of most filesystems, EXT, UFS and Reiser in particular. I didn’t realize until I was actually speaking to other people about filesystems how boring and monotonous they are.

Code
Programs that I have written to solve specific little problems.

playlist.pl - A perl script to automatically turn an iTunes playlist into an mp3 directory.
portscan.pl - A simple portscanner I wrote to learn Perl threads.
ws.pl - This script was, for the most part, originally in 2600 magazine. The printed version didn’t work very well, my version is a lot faster and some other stuff that I don’t remember. It’s well commented, interesting to look at if you like networking. It can find all webservers on the given subnet.

Wikipedia

Dec 29, 2005

If you ever talk to me in real life, you’ve undoubtedly heard me extol the virtues of Wikipedia (I just noticed that wikipaedia.org is owned by a squator, sad), the most wide reaching cache of information on the internet. Well, I really dont’ have anything particularly interesting to say, though today I learned about The Scream, Lhasa Apsos, football and heat pumps. Use Wikipedia, love it and if you do both, donate to wikipedia; they’ve currently collected about $152k, which is a lot for an interweb organization.

Nmap Tutorial

Darwin Kernel Compile

Compiling the Darwin Kernel

Prepare the Build Environment

Building the Kernel

Sources

Filesystems

Computers

Wikipedia

See Also

Recent Articles

Tags

Posts by Year