Speed of league file and HTML reports generation - Page 2

kq76 · 11-10-2008, 11:25 PM

Link to tech support thread on this topic

mikev · 11-16-2008, 11:40 PM

for reference:

7:38 pm, just clicked "create archive"

mikev · 11-17-2008, 12:25 AM

8:24 Pm

21%

mikev · 11-17-2008, 02:05 AM

10:04 pm

60%

mikev · 11-17-2008, 03:17 AM

11:16 PM

finally done!

Tarball size: 55.33 megabytes.

Nearly 4 hours to create a 55 meg tarball.

With OOTP8, this was about an hour long process.

satchel · 11-19-2008, 01:04 PM

Creating the tarball with the in-game archiver? Is this the league files tarball, or one for the reports?

mikev · 11-19-2008, 05:12 PM

Quote:

Originally Posted by satchel

Creating the tarball with the in-game archiver? Is this the league files tarball, or one for the reports?

In game archiver. League reports.

satchel · 11-19-2008, 06:09 PM

Have you tried archiving into a tarball with a third-party utility?

I also found the OOTP in-game archiver to be so slow that I never even waited for it to complete once. You can generate reports, then "copy all reports to folder" to another folder, that way you get the exact set of reports which was generated that time. Take those reports and archive them with another program and then upload them. If you want to use the in-game utility to upload them, archive them separately, then re-insert them into the game folder.

Personally, I archive and upload with third-party utilities, which are incomparably faster, but I get the impression that you want to avoid doing that.

Like I said, I tried to use the in-game method, and I didn't have the patience to sit through it even once.

Corsairs · 11-19-2008, 06:19 PM

That's an interesting tip. I never thought about doing it that way. What 3rd-party archiving program do you use? I have WinRAR but I don't think that creates tarballs. I don't suppose there's any easy way to archive the necessary league files out of game?

satchel · 11-19-2008, 10:42 PM

I use PeaZip and it is excellent. Takes a little configuration, but it's easy, and it can pack 400MB+ of HTML reports down to less than 20MB. Makes a great tarball/gzip, in the exact OOTP style.

I generate the reports, then use the "Copy files to" command to stick them in a new folder, then go to that folder and run PeaZip on them, and it Gzips up a great little tarball.

From there, you could use a third-party FTP uploading program (I use Filezilla but there are dozens out there, and Filezilla isn't especially good, in my opinion), or you could stick the tarball back into the OOTP folder that the game itself uses, and use the in-game FTP uploader. Just use the "Upload" command instead of using the "Archive and upload" combo).

I tarball and upload with third-party utilities. The in-game client was giving me the same headaches it was giving you guys, so I abandoned it right away, although I did give it a shot. I wasn't sure that doing it all outside of the game would be an equal substitute for what the game itself does, but it accomplishes the same thing and works exponentially more efficiently.

In theory, one could even create the league-file tarball outside of the game, but that's another matter, and I don't think it's necessary at all.

satchel · 11-19-2008, 10:46 PM

Quick note about PeaZip: if you're archiving a big bunch of reports like I do (400MB+ like I said above) then it is going to take a few minutes to tarball up. I have a quad-core CPU on a P38-chipset motherboard with 4GB of high-performance RAM, and it takes my machine about four or five minutes to pack the 400MB down to about 20MB. So it's not going to happen instantly; but it's going to be a heck of a lot faster than three hours.

satchel · 11-19-2008, 10:50 PM

Configure PeaZip to "Ultra" compression to get the smallest tarball. It does take longer to compress and to extract, but the time you save in uploading more than makes up for it, and I've done it dozens of times with no corruption ever.

Corsairs · 11-20-2008, 01:24 AM

I've downloaded PeaZip and intend to give this a try during tonight's sim. Now I'm not sure I know where to set the compression to "Ultra". In the "Options" tab I see settings for compression, but under the "TAR" format they're all locked for me except for the "Compress files open for writing" checkbox (which defaults to unchecked). Is this where I'm supposed to set for Ultra, and if so do you know how to make these settings modifiable?

mikev · 11-20-2008, 12:22 PM

Quote:

Originally Posted by satchel

Have you tried archiving into a tarball with a third-party utility?

I also found the OOTP in-game archiver to be so slow that I never even waited for it to complete once. You can generate reports, then "copy all reports to folder" to another folder, that way you get the exact set of reports which was generated that time. Take those reports and archive them with another program and then upload them. If you want to use the in-game utility to upload them, archive them separately, then re-insert them into the game folder.

Personally, I archive and upload with third-party utilities, which are incomparably faster, but I get the impression that you want to avoid doing that.

Like I said, I tried to use the in-game method, and I didn't have the patience to sit through it even once.

I use a third party upload, but not to archive. I'll give that a shot and see if it's any different.

satchel · 11-20-2008, 02:00 PM

Quote:

Originally Posted by Corsairs

I've downloaded PeaZip and intend to give this a try during tonight's sim. Now I'm not sure I know where to set the compression to "Ultra". In the "Options" tab I see settings for compression, but under the "TAR" format they're all locked for me except for the "Compress files open for writing" checkbox (which defaults to unchecked). Is this where I'm supposed to set for Ultra, and if so do you know how to make these settings modifiable?

In the Options tab, go to the pulldown and choose "Gzip" and check the "TAR before" option. Then you should be able to specify the "Compression: Level"

I assume that you're wanting to end up with a .tar.gz file.

Corsairs · 11-20-2008, 05:06 PM

Quote:

Originally Posted by satchel

In the Options tab, go to the pulldown and choose "Gzip" and check the "TAR before" option. Then you should be able to specify the "Compression: Level"

I assume that you're wanting to end up with a .tar.gz file.

Aha, I gotcha! Alright, I think I'm all set for tonight. I'm looking forward to this... right now anything would be a big improvement over using the in-game archiver.

satchel · 11-20-2008, 06:11 PM

Holler at me if you need to, I'll be checking my email regularly tonight. If you PM me on here, it'll email me.

Keep some of the things we've talked about in mind:

use the "copy files to" command on the second page of the "Reports" dialogue in order to get the exact reports generated on that go-round copied into a place where you can archive them easily; otherwise you'll be picking through the reports in your news/html folder to find the relevant ones (unless you want to ball up all the reports, new and old, and that's a lot of unnecessary stuff)
it's going to take PeaZip a few minutes to ball everything up; my machine is cutting-edge and it takes me about five minutes to pack everything down to 5% of original size; I don't know what your rig is like, but I could conceive of it taking anywhere from five, on up to thirty if your machine is just average, and you've got a ton of reports—just preparing you for a worst-case scenario; there is a progress bar which shows you what's going on, so you'll be able to see that it's working; using Ultra compression does drag out the ball-up process, but MB for MB, the time you save in uploading is worth it

Corsairs · 11-20-2008, 06:15 PM

That's probably a question I should ask: what kind of connection are you on? I have a slow rig but a fast internet connection and I'm wondering if I may be better off using standard compression. The final size of my reports tarball when using the in-game archiver is usually just shy of 40 MB.

mikev · 11-20-2008, 06:36 PM

The upload isn't an issue -- it's generating the tarball that is a killer.

satchel · 11-20-2008, 06:54 PM

If your machine is not particularly fast, then "normal" compression is probably a good starting point. Try it first, then experiment from there, maybe try the next-higher up one later sometime.

I suppose that the overall difference between uploading a 20MB file and a 40MB one isn't much, only a handful of minutes. It's just that when I'm dealing with archiving 400MB of data, my impulse is to pack it as tightly as possible; and the wisdom in that decision is debatable.

I've always figured that the in-game archiver is putting together the tarball at what PeaZip calls the "normal" compression level. Compare your new one's size to your recent 40MB 'ball and see what kind of a difference there is.

11-10-2008, 11:25 PM	#21
kq76 Global Moderator Join Date: Nov 2002 Posts: 11,868	Link to tech support thread on this topic __________________ My OOTP Wishlist \| My FAQ List OOTP Wiki \| Your Recommended Team Nicknames, By City (A Crowdsourced Project) For Beta/Devs: Full screen (1920x1080)

11-16-2008, 11:40 PM	#22
mikev Hall Of Famer Join Date: Dec 2004 Location: Bay Area, CA Posts: 4,014	for reference: 7:38 pm, just clicked "create archive" __________________ Global Unified Baseball Association - Vice Commish and Oakland Oaks GM

11-17-2008, 12:25 AM	#23
mikev Hall Of Famer Join Date: Dec 2004 Location: Bay Area, CA Posts: 4,014	8:24 Pm 21% __________________ Global Unified Baseball Association - Vice Commish and Oakland Oaks GM

11-17-2008, 02:05 AM	#24
mikev Hall Of Famer Join Date: Dec 2004 Location: Bay Area, CA Posts: 4,014	10:04 pm 60% __________________ Global Unified Baseball Association - Vice Commish and Oakland Oaks GM

11-17-2008, 03:17 AM	#25
mikev Hall Of Famer Join Date: Dec 2004 Location: Bay Area, CA Posts: 4,014	11:16 PM finally done! Tarball size: 55.33 megabytes. Nearly 4 hours to create a 55 meg tarball. With OOTP8, this was about an hour long process. __________________ Global Unified Baseball Association - Vice Commish and Oakland Oaks GM

11-19-2008, 01:04 PM	#26
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	Creating the tarball with the in-game archiver? Is this the league files tarball, or one for the reports?

11-19-2008, 06:09 PM	#28
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	Have you tried archiving into a tarball with a third-party utility? I also found the OOTP in-game archiver to be so slow that I never even waited for it to complete once. You can generate reports, then "copy all reports to folder" to another folder, that way you get the exact set of reports which was generated that time. Take those reports and archive them with another program and then upload them. If you want to use the in-game utility to upload them, archive them separately, then re-insert them into the game folder. Personally, I archive and upload with third-party utilities, which are incomparably faster, but I get the impression that you want to avoid doing that. Like I said, I tried to use the in-game method, and I didn't have the patience to sit through it even once.

11-19-2008, 06:19 PM	#29
Corsairs Hall Of Famer Join Date: Aug 2007 Posts: 2,360	That's an interesting tip. I never thought about doing it that way. What 3rd-party archiving program do you use? I have WinRAR but I don't think that creates tarballs. I don't suppose there's any easy way to archive the necessary league files out of game? __________________ Founder of the Planetary Extreme Baseball Alliance (PEBA) Premiere OOTP fictional league where creativity counts and imagination is your only limitation Check for openings - contact us today! Join the PEBA

11-19-2008, 10:42 PM	#30
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	I use PeaZip and it is excellent. Takes a little configuration, but it's easy, and it can pack 400MB+ of HTML reports down to less than 20MB. Makes a great tarball/gzip, in the exact OOTP style. I generate the reports, then use the "Copy files to" command to stick them in a new folder, then go to that folder and run PeaZip on them, and it Gzips up a great little tarball. From there, you could use a third-party FTP uploading program (I use Filezilla but there are dozens out there, and Filezilla isn't especially good, in my opinion), or you could stick the tarball back into the OOTP folder that the game itself uses, and use the in-game FTP uploader. Just use the "Upload" command instead of using the "Archive and upload" combo). I tarball and upload with third-party utilities. The in-game client was giving me the same headaches it was giving you guys, so I abandoned it right away, although I did give it a shot. I wasn't sure that doing it all outside of the game would be an equal substitute for what the game itself does, but it accomplishes the same thing and works exponentially more efficiently. In theory, one could even create the league-file tarball outside of the game, but that's another matter, and I don't think it's necessary at all.

11-19-2008, 10:46 PM	#31
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	Quick note about PeaZip: if you're archiving a big bunch of reports like I do (400MB+ like I said above) then it is going to take a few minutes to tarball up. I have a quad-core CPU on a P38-chipset motherboard with 4GB of high-performance RAM, and it takes my machine about four or five minutes to pack the 400MB down to about 20MB. So it's not going to happen instantly; but it's going to be a heck of a lot faster than three hours.

11-19-2008, 10:50 PM	#32
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	Configure PeaZip to "Ultra" compression to get the smallest tarball. It does take longer to compress and to extract, but the time you save in uploading more than makes up for it, and I've done it dozens of times with no corruption ever.

11-20-2008, 01:24 AM	#33
Corsairs Hall Of Famer Join Date: Aug 2007 Posts: 2,360	I've downloaded PeaZip and intend to give this a try during tonight's sim. Now I'm not sure I know where to set the compression to "Ultra". In the "Options" tab I see settings for compression, but under the "TAR" format they're all locked for me except for the "Compress files open for writing" checkbox (which defaults to unchecked). Is this where I'm supposed to set for Ultra, and if so do you know how to make these settings modifiable? __________________ Founder of the Planetary Extreme Baseball Alliance (PEBA) Premiere OOTP fictional league where creativity counts and imagination is your only limitation Check for openings - contact us today! Join the PEBA

11-20-2008, 06:11 PM	#37
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	Holler at me if you need to, I'll be checking my email regularly tonight. If you PM me on here, it'll email me. Keep some of the things we've talked about in mind: use the "copy files to" command on the second page of the "Reports" dialogue in order to get the exact reports generated on that go-round copied into a place where you can archive them easily; otherwise you'll be picking through the reports in your news/html folder to find the relevant ones (unless you want to ball up all the reports, new and old, and that's a lot of unnecessary stuff) it's going to take PeaZip a few minutes to ball everything up; my machine is cutting-edge and it takes me about five minutes to pack everything down to 5% of original size; I don't know what your rig is like, but I could conceive of it taking anywhere from five, on up to thirty if your machine is just average, and you've got a ton of reports—just preparing you for a worst-case scenario; there is a progress bar which shows you what's going on, so you'll be able to see that it's working; using Ultra compression does drag out the ball-up process, but MB for MB, the time you save in uploading is worth it

11-20-2008, 06:15 PM	#38
Corsairs Hall Of Famer Join Date: Aug 2007 Posts: 2,360	That's probably a question I should ask: what kind of connection are you on? I have a slow rig but a fast internet connection and I'm wondering if I may be better off using standard compression. The final size of my reports tarball when using the in-game archiver is usually just shy of 40 MB. __________________ Founder of the Planetary Extreme Baseball Alliance (PEBA) Premiere OOTP fictional league where creativity counts and imagination is your only limitation Check for openings - contact us today! Join the PEBA

11-20-2008, 06:36 PM	#39
mikev Hall Of Famer Join Date: Dec 2004 Location: Bay Area, CA Posts: 4,014	The upload isn't an issue -- it's generating the tarball that is a killer. __________________ Global Unified Baseball Association - Vice Commish and Oakland Oaks GM

11-20-2008, 06:54 PM	#40
satchel Hall Of Famer Join Date: Apr 2002 Location: Ft Smith Ark. USA Posts: 2,681	If your machine is not particularly fast, then "normal" compression is probably a good starting point. Try it first, then experiment from there, maybe try the next-higher up one later sometime. I suppose that the overall difference between uploading a 20MB file and a 40MB one isn't much, only a handful of minutes. It's just that when I'm dealing with archiving 400MB of data, my impulse is to pack it as tightly as possible; and the wisdom in that decision is debatable. I've always figured that the in-game archiver is putting together the tarball at what PeaZip calls the "normal" compression level. Compare your new one's size to your recent 40MB 'ball and see what kind of a difference there is.