Changes between Initial Version and Version 1 of TestingLocally


Ignore:
Timestamp:
Oct 14, 2012 2:19:40 PM (7 years ago)
Author:
trishank
Comment:

Initial import from https://www.updateframework.com/wiki/testing_uppir_locally

Legend:

Unmodified
Added
Removed
Modified
  • TestingLocally

    v1 v1  
     1= Setting up local upPIR Test Instances = 
     2 
     3This document describes how to set up local instances of an upPIR vendor, mirrors, and client for testing. 
     4 
     5First, download and unpack the tarball at the bottom of this page. 
     6 
     7Note that the files for the mirrors, client and vendor are linked between the directories, so editing one should edit the others (at least on Mac or Linux systems). This should tremendously simplify testing. 
     8 
     9To have fast XOR operations, you'll need to rebuild this part of the code. To do this you can either run python setup.py build from each directory or you can run it once and link the library over. 
     10 
     11On my laptop, I perform the following to do this: 
     12 
     13{{{ 
     14macbookpro-9c61:testing_uppir justincappos$ cd vendor/ 
     15macbookpro-9c61:vendor justincappos$ python setup.py build 
     16This setup.py script is not intended to be used for installation.  It 
     17is only constructed to build the C xordatastore.   There will be a 
     18serious version of this written later that has the usual functionality. 
     19running build 
     20running build_ext 
     21building 'fastsimplexordatastore_c' extension 
     22llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c fastsimplexordatastore.c -o /tmp/build/fastsimplexordatastore.o 
     23fastsimplexordatastore.c: In function ‘bitstring_xor_worker’: 
     24fastsimplexordatastore.c:142: warning: implicit conversion shortens 64-bit value into a 32-bit value 
     25fastsimplexordatastore.c:146: warning: implicit conversion shortens 64-bit value into a 32-bit value 
     26fastsimplexordatastore.c: In function ‘fast_XOR’: 
     27fastsimplexordatastore.c:353: warning: implicit conversion shortens 64-bit value into a 32-bit value 
     28fastsimplexordatastore.c:366: warning: implicit conversion shortens 64-bit value into a 32-bit value 
     29llvm-gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -Wl,-F. -arch i386 -arch x86_64 /tmp/build/fastsimplexordatastore.o -o /Users/justincappos/tuf/tmp/testing_uppir/vendor/fastsimplexordatastore_c.so 
     30macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror1 
     31macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror2 
     32macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../mirror3 
     33macbookpro-9c61:vendor justincappos$ ln fastsimplexordatastore_c.so ../client 
     34}}} 
     35 
     36If you're on a windows system, be sure to unpack both the zip file and the tarball. You also may need to install other software like !MinGW before it will build. 
     37 
     38If you cannot get this to work, edit uppirlib.py and uppir_mirror.py to change the lines that read: import fastsimplexordatastore to read import simplexordatastore as fastsimplexordatastore. 
     39 
     40= Setting up the files to be distributed = 
     41 
     42Now you can copy files over into a directory to be distributed. You can either have a separate directory for each mirror and the vendor (as you would actually have in practice) or share a directory. I'll share a directory called ../filestoshare. 
     43 
     44 
     45{{{ 
     46macbookpro-9c61:vendor justincappos$ mkdir ../filestoshare 
     47macbookpro-9c61:vendor justincappos$ cp LICENSE.TXT README /etc/profile /etc/man.conf ../filestoshare 
     48# I added some other files of my own 
     49macbookpro-9c61:vendor justincappos$ cp ~/seattle/trunk/www/seattle_html/* ../filestoshare/ 
     50 
     51# Now I'm ready to build a manifest... 
     52 
     53# check I'm in the vendor directory... 
     54macbookpro-9c61:vendor justincappos$ pwd 
     55/Users/justincappos/tuf/tmp/testing_uppir/vendor 
     56 
     57# build the manifest, telling it the directory to share, the block size (1024) and the IP to use (127.0.0.1) 
     58macbookpro-9c61:vendor justincappos$ python uppir_create_manifest.py ../filestoshare/ 1024 127.0.0.1 
     59Generated manifest.dat describing xordatastore with 38 1024 byte blocks 
     60}}} 
     61 
     62 
     63= Starting the vendor and mirrors = 
     64 
     65At this point, I'm ready to run the vendor. For testing purposes, it's often useful to run this in the foreground. To do this, use the --foreground flag. 
     66 
     67{{{ 
     68 
     69Justin-Capposs-MacBook-Pro:vendor justincappos$ python uppir_vendor.py --foreground 
     701320949226.18 Running in foreground.   Logging to terminal as well as: vendor.log 
     711320949226.18 ready to start vendor on 127.0.0.1:62293 
     721320949226.25 vendor service started! 
     73}}} 
     74 
     75 
     76In other terminals, you can run mirror instances as well. 
     77 
     78Change your terminal to the mirror's directory (such as ../mirror1). 
     79 
     80Each mirror will need to know where to locate the mirror files, what ports to use, a copy of the manifest file, and what ports to use. Here is an example of how to use this: 
     81 
     82 
     83{{{ 
     84Justin-Capposs-MacBook-Pro:mirror1 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62001 --foreground --mirrorroot=../filestoshare/ --httpport=61001 --retrievemanifestfrom=127.0.0.1 
     851320949550.71 Running in foreground.   Logging to terminal as well as: mirror.log 
     861320949550.75 ready to start servers! 
     871320949550.75 servers started! 
     88 
     89}}} 
     90 
     91We can run another mirror instance in a different terminal. You will need to change the mirror to another directory and change the port numbers as well. 
     92 
     93 
     94{{{ 
     95macbookpro-9c61:mirror2 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62002 --foreground --mirrorroot=../filestoshare/ --httpport=61002 --retrievemanifestfrom=127.0.0.1 
     961320949695.29 Running in foreground.   Logging to terminal as well as: mirror.log 
     971320949695.3 ready to start servers! 
     981320949695.3 servers started! 
     99}}} 
     100 
     101 
     102This process can be continued for the third mirror as well. 
     103 
     104 
     105{{{ 
     106macbookpro-9c61:mirror3 justincappos$ python uppir_mirror.py --ip=127.0.0.1 --port 62003 --foreground --mirrorroot=../filestoshare/ --httpport=61003 --retrievemanifestfrom=127.0.0.1 
     1071320950036.22 Running in foreground.   Logging to terminal as well as: mirror.log 
     1081320950036.22 ready to start servers! 
     1091320950036.22 servers started! 
     110}}} 
     111 
     112 
     113= Running an upPIR client = 
     114 
     115Now you can go and retrieve files using the uppir_client software. Open a terminal in the client directory. To retrieve the file 'profile', simply say where to retrieve the manifest from and then retrieve it. 
     116 
     117 
     118{{{ 
     119macbookpro-9c61:client justincappos$ python uppir_client.py --retrievemanifestfrom=127.0.0.1:62293 profile 
     120[u'altruism.html', u'development.html', u'education.html', u'index.html', u'LICENSE.TXT', u'man.conf', u'netadmins.html', u'profile', u'README', u'research.html', u'style.css'] 
     121profile [29] 
     122[{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}] 
     123Mirrors:  [{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}] 
     124... 
     125wrote profile 
     126}}} 
     127 
     128 
     129Once you've retrieved the manifest the first time, you can retrieve others without re-retrieving the manifest. 
     130 
     131 
     132{{{ 
     133macbookpro-9c61:client justincappos$ python uppir_client.py man.conf 
     134[u'altruism.html', u'development.html', u'education.html', u'index.html', u'LICENSE.TXT', u'man.conf', u'netadmins.html', u'profile', u'README', u'research.html', u'style.css'] 
     135man.conf [20, 21, 22, 23, 24] 
     136[{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}] 
     137Mirrors:  [{u'ip': u'127.0.0.1', u'port': 62002}, {u'ip': u'127.0.0.1', u'port': 62003}, {u'ip': u'127.0.0.1', u'port': 62001}] 
     138............... 
     139wrote man.conf 
     140}}} 
     141 
     142 
     143= Restarting the mirrors or vendor = 
     144 
     145To restart these, you can either kill them with a command like killall. You can also use CTRL-C on some OSes. On other OSes, this will raise an exception but will not exit. If this occurs, CTRL-\ should cause the program to exit. 
     146 
     147You can then re-run the code and your changes will be taken into the account. 
     148 
     149Note that if you update the files to be shared, you will need to re-build the manifest file on the vendor. Be sure to remove all of the old manifest files from the clients and mirrors.