XtremWeb-HEP 8

XtremWeb-HEP 8 introduces virtualization by permitting end users to submit their own virtual machines. XWHEP 8 raises security level to grid standards.

News

XtremWeb-HEP 8.3.0 supports certified certificates

The release 8.3.0 of XtremWeb-HEP 8 is a breakthrough in the world of Desktop Grid Computing.

For the first time, Desktop Grid security is raised to grid standards.

XtremWeb-HEP 8.0.0 Gets Support for Virtual Appliance

With XtremWeb-HEP 8 , for the first time, DG users are able to register their own Virtual Images for their preferred OS and applications and remotely deploy them on Volunteer PC.

Thanks to the embedded IBIS Smartsockets technology, users can directly and securely connect (through ssh) to the remote VM even if the PC is located behind firewall.

The convergence of Desktop Grid Computing and Cloud Computing is there.

On January 20th, 2012, the middeware is deployed at LAL for our final tests. The middleware will be released when these tests are successfully passed.

A new web site has been set up : Flying-Grid.org.

Documentation

The documentation is available here.

Bug tracking

You can glance at our trac server.

Versions

– May 7th, 2014 : XWHEP 8.3.3

• Corrections
  1. SQL statements must always comply with referential integrity
  2. LiveCD Ubuntu security hole
  3. Live CD kernel console
  4. On insertion of a new Work inside the XtremWeb-HEP database, ownerUID must NOT be null
  5. The ‘xwconfigure’ script must generate all UIDs in lowercase
  6. On insertion of a new Work inside the XtremWeb-HEP database, status must NOT be null
  7. On insertion of a new Data as work result inside the XtremWeb-HEP database, status must NOT be null
  8. Linux Live CDs for submission to XtremWeb-HEP must provision separate disks for application, input data and output data
  9. Removing a work does not delete all associated tasks
  10. update db does not work
• New functionnalities
  • n/a

– novembre 12th, 2013 : XWHEP 8.3.2

• Corrections
  1. Ibis Smartsocket Hub network
  2. sonar versionning error
  3. server generated tasks with bad ‘workUID’ in the database
  4. goups without session should be created with ‘sessionUID’ = NULL
  5. smartsocket proxy usage
  6. proxy should close streams
  7. worker debian package error
  8. Ibis Smartsockets Hub configuration
• New features
  • n/a

– Oct 7th, 2013 : XWHEP 8.3.1

• Corrections
  1. https://trac.lal.in2p3.fr/DGHEP/ticket/134 and https://trac.lal.in2p3.fr/DGHEP/ticket/135: there is now a specific installation package for the server configuration
• New features
  • n/a

– Sept 18th, 2013 : XWHEP 8.3.0

• Corrections
  1. https://trac.lal.in2p3.fr/DGHEP/ticket/120 : the server can now be certified by a Certificate Authority
  2. https://trac.lal.in2p3.fr/DGHEP/ticket/123 : after login through OpenId server the web page shows the user email address
  3. https://trac.lal.in2p3.fr/DGHEP/ticket/126 : at submission, one and only one application can be selected
  4. https://trac.lal.in2p3.fr/DGHEP/ticket/122 : server dashboard; everything is now embedded in the xwhep server using https protocol only (because recent web browser cowardly refuses to display/use any part referenced using http and not https)
  5. https://trac.lal.in2p3.fr/DGHEP/ticket/128 : a error message is sent to the client web browser on any OpenId delegation error
• New features
  • n/a

– Jun 24th, 2013 : XWHEP 8.2.3

• Corrections
  1. client (CLI and REST API) online help improved
  2. a bug corrected in session removal (https://trac.lal.in2p3.fr/DGHEP/ticket/114)
  3. a security hole corrected on user retrieval: certificate and password must not be transmitted through the network (https://trac.lal.in2p3.fr/DGHEP/ticket/109)
  4. OpenID improved: the server sends HTML status 401 on credential error (https://trac.lal.in2p3.fr/DGHEP/ticket/116)
  5. job submission corrected in the web interface; we can now select data for input (https://trac.lal.in2p3.fr/DGHEP/ticket/117)
• New features
  • n/a

– Jun 7th, 2013 : XWHEP 8.2.2

• Corrections
  1. When inserting a new session, we must force owneruid if not set
  2. When trying to insert a new object not subject to be inserted by the client (task, host), we must throw an exception and return an error to the client
  3. Some code has been rewritten in a more object oriented way. IdRpc and XWTag now have embedded memthods to create (repectively) XMLRPCCommands and TableInterfaces.
     This permits to avoid the usage of switch(idrpc) and switch(xwtag)

  4. A bug corrected on client side, in the macro parser
• New features
  1. n/a

– April 16th, 2013 : XWHEP 8.2.1

• Corrections
  1. character encoding forced to UTF-8
  2. an inter-thread synchonization bug corrected in server
• New features
  1. n/a

– Feb 26th 2013 : XWHEP 8.2.0

• Corrections
  1. LiveCD are now configured as follow
     – the login is now “vmuser” in place of “xwuser”
     – the script to create new LiveCD is in /usr/local/sbin/ (ubuntu_crealtelivecd.sh and/or sl5_createlivecd.sh)
     – the user “vmuser” is in /etc/sudoers to be able to execute /sbin/poweroff and /usr/local/sbin/ (ubuntu_crealtelivecd.sh and/or sl5_createlivecd.sh)
     – dirsetup is not allowed any more: it was a security hole
     – context.sh is executed at the end of the boot process

  2. Executor : a bug corrected regarding cleaning processes
  3. Web Interface v0.2 :
     – help migrated into the “XtremWeb-HEP User Guide”
     – introducing Web workers

  4. Virtualization contextualization nows follow the HEPiX Virtualization Working Group recommandations (http://grid.desy.de/vm/hepix/vwg/doc/html/index.shtml)
  5. bugs corrected in shell scripts
  6. code cleaned for a better usage of the Jetty library
  7. in order to integrate jopenid, several libraries upgraded
     – jetty from 6.1.7 to 8.1.8
     – log4j from 1.2.9 to 1.2.17
     – slf4j from 1.5.3 to 1.7.2
     – servlet api from 2.5 to 3.0
• New features
  1. introducing the WALLCLOCKTIME mechanism
  2. introducing RESULTREQUEST status
  3. introducing FORCENEWUID in the worker configuration file to ease deployment over clouds
  4. we can now retrieve jobs per status
  5. introducing openid usage to ease platform usage

– June 25th 2012 : XWHEP 8.1.2

– Corrections
1. on server side, the scheduling improved : hosts.incomingconnection is
   now taken into account to ensure a server like job is not sent to
   a volunteer resource where incoming connections are not allowed
2. on server side, the scheduling improved : restoring a feature from XtremWeb 1.8.0 by INRIA : “expected host” so that the end user can specify at submission time the expected volunteer resource to run the job. Note, that if the expected host does not request a job or does not match the requirements (OS, CPU…), the job is never computed.
3. the DG to SG bridge is now compatible with shared applications so that we can use VirtualBox on EGI resources
4. some bugs corrected on client side regarding application registration and task management
5. a bug corrected on temporary file usage: they are now cleaned at software exit
6. the REST interface has been completely rewritten. The <xwhep></xwhep> root element MUST be used. Examples:
   – to send a new application
   https://server/sendapp/?XWPARAM=
   – to retrieved registered applications
   https://server/getapps
7. a bug corrected in HTTP data download
8. SmartSocket usage corrected; inter node communications fully functional now; thank to the support of Jason Maassen from the Netherlands eScience Center in Amsterdam

– New features

1. the creation of new Live CD can be customized by providing several optional files:
   • a text file ‘user.packages’ may be provided to install custom packages; this file must contain a space separated packages list

   a text file ‘user.hostname’ may be provided to customize the LiveCD hostname; this file must contain the host name only

   • any package files are installed on the fly (*.rpm or *.deb, depending on the LiveCD OS)
2. the client can now be used to create SmartSockets end point, independently of any task. This may be useful to create SmartSockets tunnel from a running VM to the client PC (e.g. mount the client FS inside the running VM)
3. a new Mac OS X package “xwhep.vworker” to deploy the middleware inside a VM
4. the server contains an embedded Web service interface available through HTTPS

– May 3rd, 2012 : XWHEP 8.0.2

– Corrections
1. on server side, the scheduling improved : hosts.incomingconnection is
   now taken into account to ensure a server like job is not sent to
   a volunteer resource where incoming connections are not allowed

2. on server side, the scheduling improved : restoring a feature
   from XtremWeb 1.8.0 by INRIA : “expected host” so that the end user can
   specify at submission time the expected volunteer resource to run the job.
   Note, that if the expected host does not request a job or does not match
   the requirements (OS, CPU…), the job is never computed.

3. the DG to SG bridge is now compatible with shared applications so that
   we can use VirtualBox on EGI resources

4. some bugs corrected on client side regarding application registration and task management
5. a bug corrected on temporary file usage: they are now cleaned at software exit
6. the REST interface has been completely rewritten. The end user MUST have a valid X509 certificate.
   The root element MUST be used.
   e.g.:
   • https://server/sendapp/?XWPARAM=: to send a new application
   • https://server/getapps : to retrieved registered applications
   • https://server/getusers: to retrieved registered users
   • etc.
   • https://server/get/objectUID : to retrieved a specific registered object

– New features

1. the creation of new Live CD can be customized.
   This is done by providing several optional files:
   • a text file ‘user.packages’ may be provided to install custom packages
     this file must contain a space separated packages list

   • a text file ‘user.hostname’ may be provided to customize the LiveCD hostname
     this file must contain the host name only

   • any package files are installed on the fly (*.rpm or *.deb, depending on the LiveCD OS)
2. the client can now be used to create SmartSockets end point, independently of any task.
   This may be useful to create SmartSockets tunnel from a running VM to the client PC (e.g. mount the client FS inside the running VM)

3. a new Mac OS X package “xwhep.vworker” to deploy the middleware inside a VM

– Feb 20th, 2012 : XWHEP 8.0.1

– Corrections
1. a bug corrected in Socket Time out management: reconnection must be automatic
2. the client now compressed any data, if necessary
3. worker must not use the “connected” mode introduced in 7.4.0
4. worker and dispatcher must not use any device at all : setProperty(“java.awt.headless”, “true”)
   (See http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/)

5. a bug corrected on URI usage: ampersand are now accepted if written as “&amps;”
6. on client side, bugs corrected on XML file usage (whith “–xwxml”)
7. on client side, a bug corrected when retrieving user group

– New features

1. worker definition has new attributes:
   • shared applications : comma separated list of application names shared as volunteer sharing
   • shared library : comma separated list of library names shared as volunteer sharing
   • shared data : comma separated list of data names shared as volunteer sharing
   • incomingconnections : volunteer resoure may accept communication tunneling. The middleware does not modify the resource firewall in any manner
2. usage of IBIS Smartsockets to interconnect jobs running on worker side (to each other and to/from client)
3. a new CommandLineOption : KEEPZIP (for debug purposes)
4. work definition has new attributes:
   • readydate : when the work has been downloaded by the worker
   • datareadydate : when all data of the work has been downloaded by the worker
   • compstartdate : when the worker has started the job
   • compenddate : when the worker has ended the job
5. tasks have their own UID now (it used to be their related work one)
   this aims to keep track of all event over the platform
   we can now retrieve all run informations (all start dates and host informations per task)

6. a work can be modified by its owner or the administrator
   this is needed by SpeQulOS (EDGI JRA2)

7. we can force the cache content path for any entry
   this is needed by Bitdew/XWHEP interconnection (still under development)