Barrio Malawi - Nerds Only

Recovering Deleted files with Linux and Sleuthkit

jon — Wed, 17 May 2006 03:39:17 -0400

Ever accidentally delete a file you really needed from the recycle bin? Ever format a disk and install a new operating system only to find later that you forgot to save off some really important files first? The former recently happened to us here at the Univeristy in Mzuzu... we thought doom was upon us until we stumbled upon a open (free to change, use, and distribute) software utility called Sleuthkit (http://www.sleuthkit.org).

When we realized our disaster, we started searching Google for "data recovery", "recover my files", etc. Listings appeared for companies charging hundreds of dollars for data recovery services and software. We tried some of the trial versions of the commercial software, but nothing was able to detect our files. Finally one of us here in the office in Mzuzu stumbled upon Sleuthkit, an amazing utility that was able to recover out deleted files from a formatted disk and didn't cost us a penny!

For all those out there that do not feel comfortable removing hard disks from computers, or if all of this talk of Ubuntu and software is boring you at this stage, let me leave you with just one parting thought - if you are using a computer and discover that a file you really need has somehow been "permanently" deleted the steps you should take are:

1. Immediately stop using the computer and shut it down. When your precious file was "deleted" from the disk, it wasn't actually deleted at all. In fact all the computer did was make a note to it self that the space occupied by the file is now free to be over written. If your file has not yet been over written by new files, then recovery might possible with Sleuthkit.

2. Next, contact all the nerds you know. Tell them what has happened and tell them that you have heard of about tools like "Ubuntu", "Linux", and "Sleuthkit" that can be used to fix the problem. If the nerd's eyes light up when you mention "Ubuntu" or "Linux" you have found the right person to help.

The following are the steps we used to recover our files from the formatted disk here at the university in Mzuzu:

Remove the hard disk from the computer it was in when the files were deleted (if the disk with deleted files in on a laptop do the following: boot the laptop with a liveCD such as knoppix, mount a network drive using smbmount or nfs, use dd to write an image of the disk to the network drive, and skip to step #8 below)
Attach the disk with the "deleted" files to a computer that is running Ubuntu Linux as its operating system (I imagine your Could use a Ubuntu LiveCD here )
*** Note this computer will need to have free disk space larger than the entire size of the disk with deleted files ****
Boot up the Ubuntu computer
use System->Administration->Disks or run the command "df -h" from the command line to determine which system device number your disk being identified by. We used df -h and based on the size of our disks found that the one with the deleted files was referenced as /dev/hdb
*** Only once you are positive that you have the correct system identity of your disk should you proceed ***
Create a disk image of the disk "sudo dd if=/dev/YOUR_DISK_ID of=/home/YOUR_HOME_FOLDER/recovery_disk_image.img"
When the image of the disk with deleted files is ready, power down Ubuntu and remove the disk with the deleted files. Store it in a safe place. But do not plug it back into another computer just yet.
Turn the Ubuntu computer back on
Add the universe software repoitories to your apt sources list (see The Ubuntu Getting Started Guidi on Google for more info)
use apt or synaptic package manager to to install sleuthkit and autopsy
open a command line prompt and run "sudo autopsy"
open a web browser and go to the page http://localhost:9999/autopsy
A graphical interface appears Choose "New case" and follow the questions that the wizard asks you. When prompted for the location of the disk image enter "/home/YOUR_HOME_FOLDER/formatted_disk.img"
After sluethkit analyzes the disk image it will display a listing of the disk's partitions. Choose the partition where you deleted file was saved and click Analyze -> File Analysis
Along the left hand side of the screen you will see a search box where you can search for the deleted file by name. Enter the file name and click Search.
A list of the files matching your search text will appear. Browse the list of files and click on the hyperlink name of the file that you would like to recover
Once the file finishes loading click the Export hyperlink and wait for your web browser to prompt you to save the file
When prompted save the file
Test to make sure your “deleted” file opens correctly on another computer. Transfer the file to another computer using a CD, USB flash drive, floppy disk or the internet

Repeat steps 14 -18 to recover more files that were deleted. Using these steps we were able to recover an entire folder of files from a disk that had been formatted and reinstalled with a new version of Windows XP. Hope it works for you as well as it worked for us

Further evidence of a dotcom boom in Malawi

jon — Fri, 12 May 2006 17:18:55 -0400

Further evidence of a dotcom boom in Malawi... Last week the students of Mzuzu University went on strike because they felt that they had not been provided enough access to computers by campus administration. Unfortunately, (or fortunately depending upon how you look at it), Marissa and I were out of town at the time, in Tanzania. We have heard that the student strike was mostly peaceful. Students organized a sit-in and skipped classes. One of the days of the strike, however, things seem to have gotten a little out of hand. Students decided to burn old tires near the campus gate, so the police and some sort of weak tear gas became involved.

As a result of the strikes and demonstrations, the University administration has purchased and installed 10 brand new computers for the students.

Mzuzu University is well know throughout the country for being the most orderly of the universities. This type of unrest is very unusual here. The good news is that things, including the introduction to computer programming course that I am teaching, seem to be all back to normal now.

VSAT Community Networks

jon — Fri, 12 May 2006 17:15:47 -0400

I just read that the internet is growing faster in Africa than anywhere else in the world. Most of us here in Africa are connecting via satellite or VSAT connections which are a fairly expensive way to transmit data (though costs are rapidly dropping). While I am finding sufficient knowledge on how to purchase VSATs, I have found very little discussion or collaboration online concerning how best to maintain VSAT internet connections.

Just yesterday I received an email from a friend here in Malawi describing the experience of another friend who “successfully” installed a VSAT connection for their NGO. The NGO found that the the installers from the company they hired to install the VSAT “haven't got a clue [how to install a VSAT]. It took them over 10 days to get it running - basically they installed it and it didn't work and they couldn't work out why.” After asking help from a local internet cafe owner the NGO found “the VSAT company had supplied the wrong kind of network cable (not a cross over one) - a fairly basic mistake and they haven't apologised or anything - so we're not really very impressed.”

Poor quality technical support seems to be a common complaint of those organizations installing VSATs. Often the price for poor technical support in VSAT service contracts is very high. Organizations are usually asked to pay travel, room and board expenses for foreign technical support consultants in addition to paying fees for fixing connection problems. In the case of the University here in Mzuzu, we have found that maintaining the connection can be challenging because the network infrastructure was designed by outside consultants. Very few of the skills needed to keep the VSAT running optimally were transferred in documentation or training to the University IT staff.

Here in Mzuzu there is some consensus that organizations using VSATs could themselves improve technical support services while reducing service contract costs by collaborating online. We are interested in creating an online community to help other organizations around the world sustain and manage their VSAT internet connections. This would include: tools/guides to help organizations estimate and budget for usage costs, a tutorial collection to help build staff capacity for maintaining the VSAT and reduce dependence on outside consultants, a collection of success stories and best practices/uses for internet (giving professors blogs, etc), an online forum for technical questions to be answered by other members of VSAT community.

The other phase of our work here involves designing open source software to control, allocate and optimize network bandwidth on the VSAT (as I discussed before). I truly believe that this is the this is the other piece of the puzzle that could really fuel of fire of internet development here in Africa. User friendly software that turns old PCs into network appliances for allocating bandwidth would empower community groups, NGOs, churches, universities, and cities and allow them to collectively purchase VSAT equipment and distrubute and manage their bandwidth according each organizations contribution towards the connection cost.

I have found that the NoCat (http://www.nocat.net) project is pretty much what we need to begin allocating bandwidth based upon user logins. It is a great community networking product, but the problem is, the project stopped development sometime in 2004. Currently assessing the state of the NoCat code to see if it suits our needs and can be built upon. We are also looking into rolling out our own system.

Here is a more technical details of the community network system we are trying to create:

Place a Linux authentication gateway to filter all network traffic just before it is sent out the VSAT to the rest of the world
The authentication gateway should optionally allow unlimited local network traffic (email, file and print sharing, web browsing)
When a user tries to send data out to the world through the VSAT, the authentication gateway captures the request and the user is asked to login
After logging in network bandwidth limits are set on the gateway on a per user basis according to which network privileges the user has been granted. These are settings controlled by the network administrator.
The user is shown how much network time they have remaining in their account, how much bandwidth has been allocated to them etc.
The user is redirected to the website the originally requested before the capture. A small javascipt frame in the web browser continually checks in with the authentication gateway so that the gateway knows the user is still using the network.
Once logged in the user can send any type of network requests that are permitted by the network administrator through the VSAT
To end the VSAT network session the user closes their web browser
The authentication gateway notices that the user has not checked in for some time and closes network access to the VSAT for the user.

NoCat does most of this already, but is lacking an accounting system that would permit easy creation of network access limits for groups of users. Now to figure out if we can add it to the system... I love nerding.

Professor Saints and the nutty open source textbook

jon — Sun, 23 Apr 2006 20:44:26 -0400

I must apologize for the lack of content on the blog recently. Things have been busy in good ways... very good ways. I have been asked to teach the first ever introductory to computer science course here at Mzuzu University Department of Mathematics! Fortunately the course is only meeting twice a week, so there is plenty of time to piece together lessons and lectures. So far, the students seem to be enjoying themselves. They are very excited about having the chance to learn to program computers. Our only problem so far is that the number of computers on campus comes no where close to matching the students enthusiasm for the subject.

Agreeing to teach at the University has also opened some doors for my research. The other professors seem much more comfortable talking to another lecturer. When I was considered to be only a researcher, it seemed like the faculty here wasn't sure where how to classify me or how they could really work with me.

For our class text book we are using a very neat book called “How to Think Like a Computer Scientist” (http://www.thinkpython.com). I read it this past summer and quickly found it to be one of the best introductions to computer science that I have ever seen. The examples are clear, and topics are presented in a way that gets students writing fairly complex programs in a short amount of time. In addition to being a very high quality text book, the book is also published under an open source license which means the authors have given rights to everyone to download, print, edit, rewrite, or reproduce the text book legally in any way they choose.

The book is has three main authors – a professor from University of Chicago, a high school teacher, and a professional programmer – and claims to have received contributions from over one hundred readers who have proof-read and improved confusing sections of the book. I know that some might ask why on earth the three authors have agreed to let their work be redistributed for free without requiring any sort of monetary compensation from the users of the textbook. You are welcome to ask the authors themselves, but I think the answer might be that by opening their book – making it available for free - the authors have, in fact, benefited greatly economically. Hundreds of people have proof read and contributed to their textbook and the authors have not had to pay any of them a cent! Could you imagine paying an army of one hundred people to edit and improve a book you had written? Also, by opening their book, the authors have built names for themselves. Without waiting for a publisher's approval, these authors have published their book and watched it quickly gain in popularity in a genre where there are hundreds of competing textbooks and authors.

I took me a while to explain the concept of an open source textbook to the faculty here at the University. In Malawi, they are used to receiving “free” textbooks. Loads of old, hand-me-down text books arrive from universities in the USA and Europe each year for “free”. These textbooks are “free” because they are old and no longer wanted. My challenge here was to explain that this open source text book is “free” for a different reason. “How to Think Like a Computer Scientist” is free because the authors have decided that the most productive way they could produce a high quality, cutting edge textbook is to give it away for free and encourage students and teachers around the world to improve the text. The textbook is “free”, not because its not longer good enough for the universities of the USA and Europe... but “free”, because the authors have successfully adopted the proven open source development model (which has brought numerous high quality software products to the market) for literature.

My students, like all university students around the world, are pretty short on cash. The high quality and low cost of this text book made it the ideal textbook for our introductory computer science course here in Malawi. We have printed 5 copies ($4.00 each in our library) for a class set for the library, and the students are currently saving electronic copies of the book to their floppy disks that they carry with them. The administration here seems to also really like the idea of the open textbook. Next year, when Mzuzu University wants to get its hands on one of the latest and greatest computer science text books the University won't be dependent upon the charity of foreign universities or publishers. The updated version of “How to Think like a Computer Scientist” will be there “free” and ready for our students to download and begin reading.

As I think more about the textbook, the way in which it was produced, the impact that it is having here already... I realized more and more... open source, as model for production, clearly stands to have a enormous impact developing world.

Yet I still believe that most of us, having grown up in a copyright society, have a completely wrong impression of what open source (copyleft) development really is all about. Open source production is not some trendy new fashioned form of communism... there are real economic advantages and efficiencies at play here. The way “How to Think Like a Computer Scientist” was able to “hire” hundreds of editors for almost no money is just one example.

My current (though humble) career is another. Most of you know that I spend quite a bit of my free time and earn the majority of my income by working on open source software projects. Yes, its true, I earn money by giving my software away for “free”. But, I want people to know that I give it away, not because I am a good samaritan... I give it away because it is the best way for me to make money. I am an efficient programmer. I am a not such an efficient marketer or a lawyer. If I tried to market my software or enforce copyrights upon the code I have written, I believe that I would be wasting my time, money, and energy. I would rather spend my resources writing more creative software. By distributing my quality code for free I encourage people to use it. The more people that use it, the more likely one of two things will happen: one of the users later be willing to pay me to improve the code I have written, or some other programmer will add functionality to the code then share the changes back with me. Whether the a new client pays me to write new code, or a programmer (that I don't pay) writes a new feature that I can then sell to my clients... either way, I gain economically.

We should be clear that producers (like me) aren't the only ones that benefit in the open source production model. Buyers (my clients) can benefit as well. Usually when I am considered for hire on a software project, I tell the prospective clients that I will only take the job if I am allowed to give away the code that I write during the project for free to the open source community. Clients are always nervous about the idea at first. But there are strong economic arguments they must consider as to why it is to their benefit to pay me to give away my code for free. Sounds crazy I know... but take a client that recently hired me for a small $1000 USD dollar development project in Washington DC as an example. For that price I was able to develop some new functionality for their website. The problem was that I would soon be moving away to Malawi and no longer be able to help them maintain the new functionality. After I completed my work, we “open sourced” the code I wrote. We contributed the code to one of the major open source software projects called Drupal (software to help people manage websites). Shortly afterward, I began receiving request from around the world from other people who were interested in using it in their client's websites. Whether these people sold the code I wrote to their clients or not, it didn't matter. We let everyone who asked have the code. To every one I sent the code, I included a note said I could not promise to maintain the code after I moved away. They would have to use/sell/study the code at their own risk (this is common in open source production). Just two months before I left for Malawi, one of the “free lunch” users contacted me with the news that they were interested in taking over development of the project. They would to continue to update the code and add even more new features to it (not so “free lunch” after all).

Just last month I noticed that a new version of the code I wrote was released for everyone to use for free. There is now a team of three programmers working to maintain the code. The code is higher quality and includes even more new features than the original code i wrote. My clients in Washington DC are now free to use this new and improved code. So for the small price of their $1000 original investment my clients also now have a team of three programmers working to help maintain this new functionality for them that they do not have to pay. They get all of this because they were willing to take the risk open sourcing production of product that, in the end, proved to be valuable to other people.

And so, now I have to admit, that I really do not like the first chapter of the book “How to Think Like a computer Scientist”. I am thinking of rewriting the first chapter of the book using material from the first lecture I gave for class here entitled “How to think like a Computer”. If I do write the text, I will give it away for free to the authors to use in the next edition of the textbook. Why do I want to write a chapter for the book for “free”? Yes, there is part of me that just wants to see a better version of the book next year... but also I know that it would benefit me and my resume greatly to be able to say that I was a contributing author to one of the best computer science text books out there. If I do write a better first chapter both the book's readers, and myself stand to benefit. This is how open source production works.

Differences between internet connectivity in Colorado and Malawi

jon — Sun, 23 Apr 2006 20:37:34 -0400

Malawians, in general, are both very poor and very ambitious. I think this is why I am finding engineering in Malawi to be such exhilarating experience. So often here we are asked to do a lot with a little. Its forcing me to push the limits my creativity, my imagination, and my skills as an engineer.

I was in a meeting last week to discuss the redesign of the internet network architecture for Mzuzu University. There is a need to rethink how users are allowed to connect to the internet here in Mzuzu. Currently the network is designed much like the networks of the universities in the USA. In general, computers with connections allow users are to have unlimited access to the internet. A quick comparison of the internet connection here in Mzuzu to the last connection I used at my house in the USA will show why a different network architecture is necessary in Mzuzu:

	Mzuzu University	My Mountain House in Colorado
Bandwidth (speed of connection)	256 kbps	> 1000 kbps
Number of users the connection serves	about 500	2
Price per month	at least $300 - $400 (this could be much more, but getting the actual numbers has been tough)	$55

In English the table basically says: In Mzuzu we are trying to serve about 500 people with an internet connection that is about 5 times slower than the one broadband users have at home in the USA. This same slower connection costs us about 6-8 times what it costs home users in the states.

Again, the trick is to do a lot with a little. At home in the USA where there is a lot of cheap extra bandwidth readily available the efficiency of the design of your home network is not of much concern. If for example there is a virus sitting on your home computer sending out data via your internet connection, chances are the data the virus sends costs you very little money. Here in Malawi this is not the case. Over our satellite internet connection, bandwidth is scarce. Viruses sending out data cost us a lot of money. For many reasons of both, economics and engineering, we need to be very efficient here with how data is sent from our network out to the world via the internet.

So Mzuzu University has put together a team of myself, a volunteer from Japan and the university IT staff to design a system network architecture to use the resources of the VSAT internet connection more efficiently. We decided unanimously that we would try our best to carefully document the system we design so that it could be easily implemented by other universities who use satellites to connect to the internet.

Overall our goal is to design a system that gives as much free access to the internet as the University can afford. We want to limit users only so that they realize their internet time is expensive and should be used only in the most productive ways. By limiting the number of hours users have available for free they will respect the time available to them more and will begin to treat the internet as a limited resource.

In our first meeting we decided the requirements for the new university network. I am posting them here in hopes that some of you computer networking nerds out there might take internet in the project.

The requirements :

The network should allow users unlimited access to the local network (intra-univeristy network traffic like email , file sharing, and web browsing cost us very very little and should be allowed to happen all of the time)
Only network traffic sent via the VSAT to the rest of the world should be limited (this is the expensive network traffic)
the University has decided on set number of free hours for various groups (staff, students, administration, etc) available per week or per semester and on a pricing schedule for additional internet time. The network should track individual users use of the VSAT connection and limit their connectivity based upon the limits defined by type of user they are ( for example, a student may be given 3 hours of free connectivity per semester, while a staff member may receive 5 hours per week).
The pricing architecture should allow administration to change pricing of internet access easily and frequently
Users should be able to view their reaming time and account balances
Users should be notified with their remaining free time or account balance is low
System should allow large files and full mirrors of websites to be downloaded during off peak hours for internet usage (nights and weekends) at no charge to University network users
When power goes out the system should automatically credit one minute to users accounts and stop deducting time
new network should continue to use the web proxy cache already in place to minimize duplicate downloads
new network should include bandwidth optimization and local queuing of internet traffic to allow highly interactive traffic like Voice over internet, Video conferencing, ssh, web browsing to take priority over non-interactive uses of the internet connection like file downloads (this will also discourage large number of large files from being downloaded as they will go at slower and slower rates without affecting interactive web traffic)

Thoughts anyone?

One thousand roads and no highway...

jon — Wed, 22 Feb 2006 05:57:19 -0500

Three days into our Malawi adventure and my head is full of questions. I think that I have pieced together a reasonable view of what the internet infrastructure looks like in Malawi through various meetings at the US embassy here in Lilongwe.

It seems that in Malawi, any government office, international organization or private company who wants to connect to the internet does so by buying their own VSAT satellites. Its essentially every person for themselves, as each organization takes their chunk of capital and builds their own personal pipeline.

Its as if a village of people wanted to travel to a distant city, instead of pooling resources to build a highway that would provide faster/more reliable/efficient transportation, the people each decided to carve their own individual narrow roads to the far away city. There is an unbelieved able amount of upkeep and investment costs associated with keeping these "individual roads" or fragmented internet oasises running.

This fragmentation is frustrating to me, again, I am new here, but it seems that these VSATs are very sort-term and individualized solutions that serve very few people. They could never really be the economic catalysts to Malawi that I think the internet should be. Maybe I am naive, but It makes me wonder if some sort of resource pooling campaign couldn't be possible to link up to the nearest fiber connection here in Africa?

There is a rumor here in Malawi that there is no wire that crosses the Malawian border... not power... not electricity... not data.

I wonder where the nearest fiber optic cable is to Malawi? Who could I begin asking? Could it be streched here?