follow

#59 [Module] Suggestion - downloader module

Reported by Tom Morton 2010-01-05

- allows other modules to set up scheduled downloads or one off downloads at es_map_start

- manages dl's and then runs callback for the specified module

- you can tell it how urgent the dl is

- ensures only one dl is run per es_map_start (in order of priority)

Attachments

Comments and changes

1. #1 Tom Morton

   written 2010-01-05

   Adding some spec for xadownload module

   ref #59

   → d199166b6d0b
2. #2 Jonas Obrist

   written 2010-01-05

   I'm really in favor of a threaded/multi-processed downloader (at least as an option) since in the vast majority of cases this should provide a smoother experience for the end user.

   I agree on the idea with a scheduler tho, maybe have a look at my cheapcron.
3. #3 Tom Morton

   written 2010-01-05

   To bring our PM convo to here (and sum up so far).

   threading is a bust on Linux, which means we would have to handle that threading is generally discouraged anyway because it is totally unpredictable in python

   multiprocessing: the backport of multiprocessing lib doesn't work stably (tried it) with ES and anyway is fraught with massive massive danger

   All my testing with threading downloads has proven it is painful to get it right for everyone. Downloading at es_map_start works well for everyone in my tests and is the best, IMO, experience.

   Wget support is a good idea - and that should be included along with instruction on how to install wget

   Cheapcron looks just the ticket!
4. #4 Jonas Obrist

   written 2010-01-05

   With my module there's no need for threading in almost all cases on linux. Since I'd say the vast majority of linux systems have wget. What I mean with multiprocessing is wget, not python-multiprocessing.
5. #5 Tom Morton

   written 2010-01-05

   Ok I'm with you.

   Wget is standard on most distros. But isn't guaranteed as it may well be removed or restricted. But we can handle that.

   Rather than degrading to threaded requests I think es-map_start is a safer bet. I think this needs more input.

   The thing is that there isn't currently an application where downloads are required instantly. Maybe gauth is urgent. Even in websync we could just do end map when the cmd is run. Indeed that would be better.
6. #6 Jonas Obrist

   written 2010-01-05

   we can check for wget, curl, lynx and all the other tools for downloading websites. Also if you read my code, it actually checks if wget is installed, it doesn't just assume it's installed on linux.
7. #7 Tom Morton

   written 2010-03-04

   • Changed milestone from scrum to future-maybe.
8. #8 Tom Morton

   written 2010-04-28

   • Changed status from new to wontfix.
   • Changed version from 1.1.0.x to 2.0.0.x.

Verification: Please write the text from the image in the box (letters only)

Is that you, Humanoid? Is this me?

• Cancel