wiki:TestingLocally

Setting up local upPIR Test Instances

This document describes how to set up local instances of an upPIR vendor, mirrors, and client for testing.

First, download and unpack the tarball from here: https://uppir.poly.edu/projects/project/browser/uppir-prerelease-alpha.tgz.

You should copy and paste the files to different directories for each participant: e.g. "vendor" dir where you run the vendor from, "mirror1" dir, "mirror2" dir, "mirror3" dir, ... , and "client" dir.

To have fast XOR operations, you'll need to rebuild this part of the code. To do this you can either run python setup.py build from each directory or you can run it once and link the library over.

On my laptop, I perform the following to do this:

macbookpro-9c61:testing_uppir justincappos$ cd vendor/
macbookpro-9c61:vendor justincappos$ python setup.py build
This setup.py script is not intended to be used for installation.  It
is only constructed to build the C xordatastore.   There will be a
serious version of this written later that has the usual functionality.
running build
running build_ext
building 'fastsimplexordatastore_c' extension
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c fastsimplexordatastore.c -o /tmp/build/fastsimplexordatastore.o
fastsimplexordatastore.c: In function ‘bitstring_xor_worker’:
fastsimplexordatastore.c:142: warning: implicit conversion shortens 64-bit value into a 32-bit value
fastsimplexordatastore.c:146: warning: implicit conversion shortens 64-bit value into a 32-bit value
fastsimplexordatastore.c: In function ‘fast_XOR’:
fastsimplexordatastore.c:353: warning: implicit conversion shortens 64-bit value into a 32-bit value
fastsimplexordatastore.c:366: warning: implicit conversion shortens 64-bit value into a 32-bit value
llvm-gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -Wl,-F. -arch i386 -arch x86_64 /tmp/build/fastsimplexordatastore.o -o /Users/justincappos/tuf/tmp/testing_uppir/vendor/fastsimplexordatastore_c.so
macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror1
macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror2
macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror3
macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../client

If you're on a windows system, be sure to unpack both the zip file and the tarball. You also may need to install other software like !MinGW before it will build.

If you cannot get this to work, edit uppirlib.py and uppir_mirror.py to change the lines that read: import fastsimplexordatastore to read import simplexordatastore as fastsimplexordatastore.

Setting up the files to be distributed

Now you can copy files over into a directory to be distributed. You can either have a separate directory for each mirror and the vendor (as you would actually have in practice) or share a directory. I'll share a directory called ../filestoshare.

macbookpro-9c61:vendor justincappos$ mkdir ../filestoshare
macbookpro-9c61:vendor justincappos$ cp LICENSE.TXT README /etc/profile /etc/man.conf ../filestoshare
# I added some other files of my own
macbookpro-9c61:vendor justincappos$ cp ~/seattle/trunk/www/seattle_html/* ../filestoshare/

# Now I'm ready to build a manifest...

# check I'm in the vendor directory...
macbookpro-9c61:vendor justincappos$ pwd
/Users/justincappos/tuf/tmp/testing_uppir/vendor

# build the manifest, telling it the directory to share, the block size (1024) and the IP to use (127.0.0.1)
macbookpro-9c61:vendor justincappos$ python uppir_create_manifest.py ../filestoshare/ 1024 127.0.0.1
Generated manifest.dat describing xordatastore with 38 1024 byte blocks

Starting the vendor and mirrors

At this point, I'm ready to run the vendor. For testing purposes, it's often useful to run this in the foreground. To do this, use the --foreground flag.

Justin-Capposs-MacBook-Pro:vendor justincappos$ python uppir_vendor.py --foreground
1320949226.18 Running in foreground.   Logging to terminal as well as: vendor.log
1320949226.18 ready to start vendor on 127.0.0.1:62293
1320949226.25 vendor service started!

In other terminals, you can run mirror instances as well.

Change your terminal to the mirror's directory (such as ../mirror1).

Each mirror will need to know where to locate the mirror files, what ports to use, a copy of the manifest file, and what ports to use. Here is an example of how to use this:

Justin-Capposs-MacBook-Pro:mirror1 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62001 --foreground --mirrorroot=../filestoshare/ --httpport=61001 --retrievemanifestfrom=127.0.0.1
1320949550.71 Running in foreground.   Logging to terminal as well as: mirror.log
1320949550.75 ready to start servers!
1320949550.75 servers started!

We can run another mirror instance in a different terminal. You will need to change the mirror to another directory and change the port numbers as well.

macbookpro-9c61:mirror2 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62002 --foreground --mirrorroot=../filestoshare/ --httpport=61002 --retrievemanifestfrom=127.0.0.1
1320949695.29 Running in foreground.   Logging to terminal as well as: mirror.log
1320949695.3 ready to start servers!
1320949695.3 servers started!

This process can be continued for the third mirror as well.

macbookpro-9c61:mirror3 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62003 --foreground --mirrorroot=../filestoshare/ --httpport=61003 --retrievemanifestfrom=127.0.0.1
1320950036.22 Running in foreground.   Logging to terminal as well as: mirror.log
1320950036.22 ready to start servers!
1320950036.22 servers started!

Running an upPIR client

Now you can go and retrieve files using the uppir_client software. Open a terminal in the client directory. To retrieve the file 'profile', simply say where to retrieve the manifest from and then retrieve it.

macbookpro-9c61:client justincappos$ python uppir_client.py --retrievemanifestfrom=127.0.0.1:62293 profile
[u'altruism.html', u'development.html', u'education.html', u'index.html', u'LICENSE.TXT', u'man.conf', u'netadmins.html', u'profile', u'README', u'research.html', u'style.css']
profile [29]
[{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}]
Mirrors:  [{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}]
...
wrote profile

Once you've retrieved the manifest the first time, you can retrieve others without re-retrieving the manifest.

macbookpro-9c61:client justincappos$ python uppir_client.py man.conf
[u'altruism.html', u'development.html', u'education.html', u'index.html', u'LICENSE.TXT', u'man.conf', u'netadmins.html', u'profile', u'README', u'research.html', u'style.css']
man.conf [20, 21, 22, 23, 24]
[{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}]
Mirrors:  [{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}]
...............
wrote man.conf

Restarting the mirrors or vendor

To restart these, you can either kill them with a command like killall. You can also use CTRL-C on some OSes. On other OSes, this will raise an exception but will not exit. If this occurs, CTRL-\ should cause the program to exit.

You can then re-run the code and your changes will be taken into the account.

Note that if you update the files to be shared, you will need to re-build the manifest file on the vendor. Be sure to remove all of the old manifest files from the clients and mirrors.

Last modified 4 years ago Last modified on Nov 11, 2013 1:42:36 PM