Jump to content

Recommended Posts

  • 4 weeks later...
  • 2 years later...
Posted (edited)

I replaced the unpacker with tar.exe (a standard component in Windows 10-11, located in System32).
This made opening ePUB files approximately 2-3 times faster, and eliminated the need to copy them to TEMP, rename them to ZIP, and then delete them.

modified_files.zip

Edited by nbb1967
Posted
On 3/15/2024 at 4:07 AM, Andreik said:

My next project it's an application to manage all these ebup books. I have a large number of books and it's a pain to find what I am looking for even if I place them in different categories. I wonder how you and @argumentum (with his impressive books collection) manage to do that.

 

EDIT by Jos: I have split off the discussion about Clibre-web and others into its own thread in Chat to keep this example clean.

Having a fresh look at this project, I thought for a moment I had forgotten to reply, and the Clibre instead of Calibre spelling error probably didn't help. :lol: 

Anyway, with a second look, first had been quick, I realized I had replied and it had been relocated to another thread.

With my first look I was much more about downloading the script and giving it a try, as I can't recall doing so back when I had replied ... maybe that got sidetracked when some of the discussion got relocated. In any case, I have given it a try now, and it works well so far, if somewhat basic and limited. Some of my thoughts follow, and sorry for their lateness.

As I think @mLipok mentioned, there appears to be no page skipping, just what you can loosely call chapters, but really sections ... at least in the ebook I tried. We of course need page skipping, because scrolling is not really a desirable option in an ereader. Related to that, the jump to first chapter button, just really jumps to the start of the ebook (cover), not the first chapter.

The other thing for me, was I had to drag the window size to full screen height to see the cover in full, and it still wasn't quite enough ... almost, but no banana.

Background color is very important to be able to change, because white can be too bright. At the very least you would probably want off-white.

 Anyway, as you said, this is still in early development.

However, are you still working on this very promising project?

I am tempted to have a look at the code myself, and see what might be possible.

I think it was @TheDcoder that asked whether you are rendering the epub in embedded IE, and I don't think you ever replied? If it is just in IE, then I imagine other features are going to be problematical. So because calibre offers an independent viewer app, that mostly does all I want, I am feeling like it is probably not worth reinventing the wheel. That said, this is still is a very interesting project. :) 

P.S. I should be completely honest here, and state that it is extremely rare for me to read any kind of ebook on a PC. I either read using an e-ink reader or tablet or on the rare occasion for short periods, a phone. So all I use the calibre viewer program for, is checking an ebook appears to work properly, or to maybe look up something, or to grab an advertised free ebook or join up for a newsletter. So in other words I like the potential of your project, but are not sure where or how I personally would use it. :) 

P.S.S. Don't let some of my words discourage you. It is always good to have choices. You even have the potential of an epub editor with this project. That said, I use the wonderful free Sigil as my epub editor or the very similar editor built into calibre.

Make sure brain is in gear before opening mouth!
Remember, what is not said, can be just as important as what is said.

Spoiler

What is the Secret Key? Life is like a Donut

If I put effort into communication, I expect you to read properly & fully, or just not comment.
Ignoring those who try to divert conversation with irrelevancies.
If I'm intent on insulting you or being rude, I will be obvious, not ambiguous about it.
I'm only big and bad, to those who have an over-active imagination.

I may have the Artistic Liesense ;) to disagree with you. TheSaint's Toolbox (be advised many downloads are not working due to ISP screwup with my storage)

userbar.png

  • 3 weeks later...
Posted


https://github.com/dev-nutu/ePUB-Reader

  

3 hours ago, Andreik said:

With all the AI vibe coding I am pretty sure anyone can get in no time a better ePUB reader then what can be done using native IE features. This was a fun project in the programming era, now it's just history.  :tv_happy:

Yeap.
 

Quote

 

Signature beginning:
* Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
* ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Code * for other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API * ErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 *

 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF * SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane * 

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: * Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler * IE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related: * How to get reference to PDF object embeded in IE * IE on Windows 11 * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuff * OnHungApp handler * Avoid "AutoIt Error" message box in unknown errors  * HTML editor * 

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2023-04-24

Posted (edited)

What would be cooler I guess, seeing as there is no shortage of EPUB viewers out there, is a program that can decouple a single HTML with Base64 data into a standard web page set of file & folder, which you can then import into the wonderful free SIGIL epub editor program ... to then create an EPUB file from.

For quite a while now (years), I save all web pages I desire a copy of, to a single .html file, with Base64 images etc. Sometimes they have been an ebook web page. But alas, SIGIL gives an error when trying to import. I've never come across a program that fully reverses such a single html file, back to its file & folder pair state. In fact, I haven't found a program to convert a local html pair to the single file state either. The best I have come up with is to create a simple html file, with images that have been converted to Base64. I did that once for a tutorial.

Of course, the best solution, would be for SIGIL to support such natively or via a plugin. I kind of find it weird that they don't already.

P.S. I use the browser addon, Save Page WE, to create the single html file download. For me, it has always worked better than the other alternatives out there.

Edited by TheSaint

Make sure brain is in gear before opening mouth!
Remember, what is not said, can be just as important as what is said.

Spoiler

What is the Secret Key? Life is like a Donut

If I put effort into communication, I expect you to read properly & fully, or just not comment.
Ignoring those who try to divert conversation with irrelevancies.
If I'm intent on insulting you or being rude, I will be obvious, not ambiguous about it.
I'm only big and bad, to those who have an over-active imagination.

I may have the Artistic Liesense ;) to disagree with you. TheSaint's Toolbox (be advised many downloads are not working due to ISP screwup with my storage)

userbar.png

Posted
6 minutes ago, TheSaint said:

a program that can decouple a single HTML with Base64 data into a standard web page set of file & folder

This is a trivial task, just loop over all the media tags and extract the data and replace them with relative paths after saving them to the disk :lol:

The only complex part in this script is parsing the HTML, and even that can be done fairly trivially with StringSplit, which I heard @TheSaint is an expert in :D

Posted
3 minutes ago, TheDcoder said:

This is a trivial task, just loop over all the media tags and extract the data and replace them with relative paths after saving them to the disk

Bud for sure, I could likely reverse the basic elements of pure text (which doesn't need converting) and image files (that do). But it is all the other html crap (CSS, JavaScript, etc) that I am not well versed in, and it would only take one such to make SIGIL abort I imagine. I am a very basic HTML coder, never got into the fancy stuff, that I see most web pages bloated with these days.

7 minutes ago, TheDcoder said:

The only complex part in this script is parsing the HTML, and even that can be done fairly trivially with StringSplit, which I heard @TheSaint is an expert in

Very funny ... not. :P 

Trivial to you maybe, and I would certainly be using StringSplit for image file data ... but that's the easy bit.

I could slip into a long rant here, about the state of web pages these days, but for the sake of everyone reading this, I will resist.

P.S. This could be just the project for you bud, where you can show your mettle, and join me in my the creative world. :lol: 

Make sure brain is in gear before opening mouth!
Remember, what is not said, can be just as important as what is said.

Spoiler

What is the Secret Key? Life is like a Donut

If I put effort into communication, I expect you to read properly & fully, or just not comment.
Ignoring those who try to divert conversation with irrelevancies.
If I'm intent on insulting you or being rude, I will be obvious, not ambiguous about it.
I'm only big and bad, to those who have an over-active imagination.

I may have the Artistic Liesense ;) to disagree with you. TheSaint's Toolbox (be advised many downloads are not working due to ISP screwup with my storage)

userbar.png

Posted
10 minutes ago, TheSaint said:

But it is all the other html crap (CSS, JavaScript, etc) that I am not well versed in, and it would only take one such to make SIGIL abort I imagine.

Well bud, as long as the input is standard the output would be standard too, we are merely extracting the assets without changing any of the other properties :)

The question whether SIGIL would accept it or not is a different one, and I imagine there's a good chance it won't because it's a dated program, so really it's not just about converting a standalone HTML to a folder based one.

14 minutes ago, TheSaint said:

P.S. This could be just the project for you bud, where you can show your mettle, and join me in my the creative world. :lol: 

If we're only talking about the bit I addressed, then it's a single prompt job for ChatGPT or other LLMs. But like I said, your goal requires doing more research on what Sigil accepts and then dealing with all that.

Posted

There is a laptop that is about to come out that it will create the applications that you need on the fly. 
So there is no need to buy anything as far as software because this thing will just create it for you. 
Meaning that if be considerate and don't kill trees was something of importance, now with this type of approach it will kill trees, and towns, so, yeah.

Note to self: Fully wake up before posting. 

Follow the link to my code contribution ( and other things too ).
FAQ - Please Read Before Posting  image.gif.922e3a93535f431de08b31ee669cc446.gif
autoit_scripter_blue_userbar.png

Posted (edited)
3 hours ago, TheDcoder said:

there's a good chance it won't because it's a dated program

Where did you get that notion from? The latest release was only last week, and it supports the latest EPUB standard.

3 hours ago, TheDcoder said:

Well bud, as long as the input is standard the output would be standard too, we are merely extracting the assets without changing any of the other properties

From my recollection some of those Base64 assets are the very things I mentioned. I know how to deal with images, but nothing beyond that.

SIGIL accepts normal html files to create ebooks from, though perhaps not something really fancy, but that's not what I am dealing with. It seems to me it is the Base64 state that is the problem.

So it seems likely to me, that all I need to do is replace all the Base64 elements with their normal html variant.

Edited by TheSaint

Make sure brain is in gear before opening mouth!
Remember, what is not said, can be just as important as what is said.

Spoiler

What is the Secret Key? Life is like a Donut

If I put effort into communication, I expect you to read properly & fully, or just not comment.
Ignoring those who try to divert conversation with irrelevancies.
If I'm intent on insulting you or being rude, I will be obvious, not ambiguous about it.
I'm only big and bad, to those who have an over-active imagination.

I may have the Artistic Liesense ;) to disagree with you. TheSaint's Toolbox (be advised many downloads are not working due to ISP screwup with my storage)

userbar.png

Posted (edited)
19 minutes ago, ioa747 said:

You might be interested  SingleFile

Nope definitely not. Tried that a few times in the past, and it just gave me grief.

Save Page WE by comparison is much better.

In any case, my issue is the Base64 aspect, which would be much the same for both addons, and any others like them.

So thanks for the suggestion, but I already have a lot of these single file html files already, and no new addon will likely help with them, not if it cannot convert local files.

Edited by TheSaint

Make sure brain is in gear before opening mouth!
Remember, what is not said, can be just as important as what is said.

Spoiler

What is the Secret Key? Life is like a Donut

If I put effort into communication, I expect you to read properly & fully, or just not comment.
Ignoring those who try to divert conversation with irrelevancies.
If I'm intent on insulting you or being rude, I will be obvious, not ambiguous about it.
I'm only big and bad, to those who have an over-active imagination.

I may have the Artistic Liesense ;) to disagree with you. TheSaint's Toolbox (be advised many downloads are not working due to ISP screwup with my storage)

userbar.png

Posted (edited)
1 hour ago, TheSaint said:

Where did you get that notion from? The latest release was only last week, and it supports the latest EPUB standard.

Nevermind, I was being dumb, should've check it before I assumed stuff. If it does accept the latest standards then I see no reason why a simple script like I suggested wouldn't do the trick, it's a one prompt job. And I decided to put my new Claude account to use and gave it this prompt to Sonnet 5.5 Extra:

Quote

Write a clean single file Python script (which is compatible with Windows) to take a single HTML file as input and extract all of the embedded media elements encoded in base64 URIs and create a new HTML file with the same content except with the media source links replace with local relative paths, and put all of the output in a folder defaulting to the same name as the input HTML file. Make no mistakes.

And it spat out the following script:

Quote

I'll write the script, then run it against a set of tricky HTML samples (wrapped base64, srcset, CSS url(), CRLF, odd encodings) before handing it over.

The script compiles cleanly. Next I'm running the edge-case test suite: wrapped base64, srcset, CSS url(), CRLF and odd encodings, and the CLI guards.

extract_base64_media.py is ready: one file, standard library only, Python 3.8+. I tested it on Linux against a 30-case sample HTML file, and all checks passed. I couldn't run it on Windows, but it only uses pathlib, binary file I/O and no shell calls.

python extract_base64_media.py page.html
python extract_base64_media.py page.html -o some_folder

By default it creates a folder named after the input (page) next to the input file. Inside are the rewritten HTML, with the same file name as the original, and a media subfolder with the extracted files. Links become relative paths like media/image_001.png, and the input file is never modified.

  • What it extracts: It extracts image, audio and video base64 data URIs wherever they appear: src, srcset, poster, href, CSS url(...) in <style> blocks and style attributes, and so on. I treated "media" as images, audio and video, so fonts and PDFs are left alone unless you add --all-types.
  • Everything else is unchanged: Only the data URIs are replaced. Line endings, encoding and whitespace stay exactly as they were, and extracted files are byte-identical to what was embedded.
  • Duplicates: Identical media is saved once and shared by every reference.
  • Bad data: A data URI that isn't valid base64 is left in place, with a warning giving its line number. Anything it skipped is also counted in the summary it prints.
  • Safety checks: It refuses to overwrite the input file, for example with -o .. It also rejects UTF-16/32 HTML with a clear error.

If you want a different layout, such as media files next to the HTML or a different default folder location, that's a small change.

#!/usr/bin/env python3
"""Extract base64-embedded media from a single HTML file.

Finds every image / audio / video that is embedded in the HTML as a base64
``data:`` URI, saves each one as a real file, and writes a copy of the HTML in
which each data URI is replaced by a relative path to the saved file.

Output layout (the folder defaults to a folder named after the input file,
created next to it)::

    page/
        page.html            <- rewritten copy of page.html
        media/
            image_001.png
            image_002.jpg
            video_001.mp4
            ...

Usage::

    python extract_base64_media.py page.html
    python extract_base64_media.py page.html -o my_output_folder
    python extract_base64_media.py page.html --all-types

Behaviour:
  * Covers every place a data URI can appear: src, srcset, poster, href,
    xlink:href, data, CSS url(...) in <style> blocks and style attributes, etc.
  * The file is processed as raw bytes, so everything except the replaced data
    URIs (encoding, BOM, line endings, whitespace) is preserved exactly.
  * Identical media is saved only once and shared by all references.
  * A data URI that cannot be decoded is left untouched and reported.
  * By default only image/*, audio/* and video/* are extracted; --all-types
    also extracts other base64 data URIs (fonts, PDFs, ...).

Requires Python 3.8+ and nothing outside the standard library.
"""

from __future__ import annotations

import argparse
import base64
import codecs
import hashlib
import html
import mimetypes
import os
import re
import sys
import urllib.parse
from pathlib import Path
from typing import Dict, List, Optional, Tuple

MEDIA_DIR_NAME = "media"
MEDIA_TOP_LEVEL_TYPES = frozenset({"image", "audio", "video"})
FILE_NAME_CATEGORIES = MEDIA_TOP_LEVEL_TYPES | {"font"}

MIME_EXTENSIONS = {
    # images
    "image/png": ".png",
    "image/apng": ".apng",
    "image/jpeg": ".jpg",
    "image/jpg": ".jpg",
    "image/pjpeg": ".jpg",
    "image/gif": ".gif",
    "image/webp": ".webp",
    "image/avif": ".avif",
    "image/svg+xml": ".svg",
    "image/bmp": ".bmp",
    "image/x-ms-bmp": ".bmp",
    "image/x-icon": ".ico",
    "image/vnd.microsoft.icon": ".ico",
    "image/tiff": ".tiff",
    "image/heic": ".heic",
    "image/heif": ".heif",
    "image/jxl": ".jxl",
    # audio
    "audio/mpeg": ".mp3",
    "audio/mp3": ".mp3",
    "audio/ogg": ".ogg",
    "audio/opus": ".opus",
    "audio/wav": ".wav",
    "audio/x-wav": ".wav",
    "audio/wave": ".wav",
    "audio/vnd.wave": ".wav",
    "audio/webm": ".weba",
    "audio/aac": ".aac",
    "audio/mp4": ".m4a",
    "audio/x-m4a": ".m4a",
    "audio/flac": ".flac",
    "audio/x-flac": ".flac",
    "audio/midi": ".mid",
    "audio/x-midi": ".mid",
    # video
    "video/mp4": ".mp4",
    "video/webm": ".webm",
    "video/ogg": ".ogv",
    "video/quicktime": ".mov",
    "video/x-msvideo": ".avi",
    "video/x-matroska": ".mkv",
    "video/mpeg": ".mpeg",
    "video/3gpp": ".3gp",
    "video/x-m4v": ".m4v",
    # non-media types (only used with --all-types)
    "application/pdf": ".pdf",
    "application/ogg": ".ogg",
    "font/woff": ".woff",
    "font/woff2": ".woff2",
    "font/ttf": ".ttf",
    "font/otf": ".otf",
    "application/font-woff": ".woff",
    "application/x-font-ttf": ".ttf",
}

# Characters that may legitimately follow the end of a data URI. Anything else
# means the URI was not recognised cleanly, so it is left untouched.
TERMINATORS = frozenset(" \t\r\n\f\"'`)<>,;\\&]}|")

_FLAGS = re.IGNORECASE | re.ASCII  # ASCII: keep \w and \s from matching Latin-1 junk

# "data:<type>/<subtype>[;parameter...];base64," (case-insensitive, and the
# scheme must not be the tail of a longer word such as "metadata:").
DATA_URI_HEAD = re.compile(
    r"(?<![A-Za-z0-9_-])data:"
    r"(?P<mime>[\w.+-]+/[\w.+-]+)"
    r"(?:;(?!base64,)[^;,\"'\s<>()]*)*"
    r";base64,",
    _FLAGS,
)

# What may follow "base64,". Besides the base64 alphabet, a few percent-escapes
# and numeric character references (+ / = and whitespace) are accepted because
# browsers decode them before decoding the base64.
_ESCAPES = r"%(?:2b|2f|3d|0a|0d|09|20)|&#(?:43|47|61|9|10|13|x2b|x2f|x3d|x9|xa|xd);"
# Normal context: the payload is one unbroken run (stops at whitespace).
PAYLOAD_STRICT = re.compile(r"(?:[A-Za-z0-9+/=]+|" + _ESCAPES + r")*", _FLAGS)
# Quoted-attribute context: whitespace/newlines inside the value are ignored.
PAYLOAD_WRAPPED = re.compile(r"(?:[A-Za-z0-9+/=\s]+|" + _ESCAPES + r")*", _FLAGS)

_ATTRIBUTE_NAME_BEFORE = re.compile(r"([^\s\"'=<>/`]+)\s*=\s*$", re.ASCII)
_ASCII_WHITESPACE = re.compile(r"[ \t\n\r\f]+")
_BASE64_BODY = re.compile(r"[A-Za-z0-9+/]*")

UNSUPPORTED_BOMS = (
    codecs.BOM_UTF32_LE,
    codecs.BOM_UTF32_BE,
    codecs.BOM_UTF16_LE,
    codecs.BOM_UTF16_BE,
)


def decode_base64(raw: str) -> bytes:
    """Decode the text after ``base64,`` the way a browser does.

    Raises ValueError when the data is not valid base64.
    """
    text = urllib.parse.unquote(html.unescape(raw))
    text = _ASCII_WHITESPACE.sub("", text)
    if len(text) % 4 == 0:
        if text.endswith("=="):
            text = text[:-2]
        elif text.endswith("="):
            text = text[:-1]
    if len(text) % 4 == 1:
        raise ValueError("invalid base64 length")
    if not _BASE64_BODY.fullmatch(text):
        raise ValueError("invalid base64 characters")
    return base64.b64decode(text + "=" * (-len(text) % 4), validate=True)


def extension_for(mime: str) -> str:
    """Return a file extension (with leading dot) for a lower-case MIME type."""
    known = MIME_EXTENSIONS.get(mime)
    if known:
        return known
    guessed = mimetypes.guess_extension(mime)
    if guessed and re.fullmatch(r"\.[A-Za-z0-9]{1,10}", guessed):
        return guessed.lower()
    subtype = mime.split("/", 1)[1].split("+", 1)[0]
    if subtype.startswith("x-"):
        subtype = subtype[2:]
    subtype = re.sub(r"[^a-z0-9]", "", subtype)[:10]
    return "." + subtype if subtype else ".bin"


def category_for(mime: str) -> str:
    """Prefix used in saved file names: image, audio, video, font or file."""
    top_level = mime.split("/", 1)[0]
    return top_level if top_level in FILE_NAME_CATEGORIES else "file"


def is_wrapped_context(text: str, start: int) -> bool:
    """True if the data URI is the entire value of a quoted, non-srcset attribute.

    Only there may the base64 be split across lines or padded with whitespace.
    In srcset (and in unquoted values or CSS url()) whitespace ends the URI.
    """
    if start == 0 or text[start - 1] not in "\"'":
        return False
    match = _ATTRIBUTE_NAME_BEFORE.search(text, max(0, start - 256), start - 1)
    if match is None:
        return False
    return not match.group(1).lower().endswith("srcset")


class MediaExtractor:
    """Rewrites HTML text, saving each embedded media file under ``media_dir``."""

    def __init__(self, media_dir: Path, all_types: bool = False) -> None:
        self.media_dir = media_dir
        self.all_types = all_types
        self._saved: Dict[Tuple[str, str], str] = {}  # (sha256, ext) -> relative URL
        self._counters: Dict[str, int] = {}
        self.replaced = 0   # data URIs replaced by a relative path
        self.skipped = 0    # data URIs that could not be decoded
        self.ignored = 0    # non-media data URIs left alone
        self.files_written = 0
        self.bytes_written = 0

    def rewrite(self, text: str) -> str:
        parts: List[str] = []
        copied_to = 0   # text before this index is already in `parts`
        search_from = 0
        while True:
            head = DATA_URI_HEAD.search(text, search_from)
            if head is None:
                break
            start, payload_start = head.start(), head.end()
            mime = head.group("mime").lower()

            payload_re = PAYLOAD_WRAPPED if is_wrapped_context(text, start) else PAYLOAD_STRICT
            end = payload_re.match(text, payload_start).end()
            search_from = end

            if not self.all_types and mime.split("/", 1)[0] not in MEDIA_TOP_LEVEL_TYPES:
                self.ignored += 1
                continue
            following = text[end:end + 1]
            if following and following not in TERMINATORS:
                self._skip(text, start, "unexpected character after the data")
                continue
            try:
                data = decode_base64(text[payload_start:end])
            except ValueError as exc:
                self._skip(text, start, str(exc))
                continue
            if not data:
                self._skip(text, start, "empty data")
                continue

            parts.append(text[copied_to:start])
            parts.append(self._save(mime, data))
            copied_to = end
            self.replaced += 1
        parts.append(text[copied_to:])
        return "".join(parts)

    def _save(self, mime: str, data: bytes) -> str:
        extension = extension_for(mime)
        key = (hashlib.sha256(data).hexdigest(), extension)
        url = self._saved.get(key)
        if url is None:
            category = category_for(mime)
            number = self._counters.get(category, 0) + 1
            self._counters[category] = number
            name = "{}_{:03d}{}".format(category, number, extension)
            self.media_dir.mkdir(parents=True, exist_ok=True)
            (self.media_dir / name).write_bytes(data)
            url = "{}/{}".format(MEDIA_DIR_NAME, name)
            self._saved[key] = url
            self.files_written += 1
            self.bytes_written += len(data)
        return url

    def _skip(self, text: str, offset: int, reason: str) -> None:
        line = text.count("\n", 0, offset) + 1
        print("warning: left a data URI on line {} untouched ({})".format(line, reason),
              file=sys.stderr)
        self.skipped += 1


def format_size(size: float) -> str:
    if size < 1024:
        return "{:d} B".format(int(size))
    for unit in ("KB", "MB", "GB"):
        size /= 1024.0
        if size < 1024 or unit == "GB":
            return "{:.1f} {}".format(size, unit)
    raise AssertionError("unreachable")


def fail(message: str) -> int:
    print("error: " + message, file=sys.stderr)
    return 1


def parse_args(argv: Optional[List[str]]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Extract base64-embedded media (images, audio, video) from an HTML "
                    "file and rewrite the HTML to use local relative paths.",
    )
    parser.add_argument("input", help="path to the input HTML file")
    parser.add_argument(
        "-o", "--output",
        help="output folder (default: a folder named after the input file, "
             "created next to it)",
    )
    parser.add_argument(
        "--all-types", action="store_true",
        help="also extract non-media base64 data URIs such as fonts and PDFs",
    )
    return parser.parse_args(argv)


def main(argv: Optional[List[str]] = None) -> int:
    # Never crash on unprintable characters in a Windows console / redirected output.
    for stream in (sys.stdout, sys.stderr):
        if hasattr(stream, "reconfigure"):
            stream.reconfigure(errors="replace")

    args = parse_args(argv)

    input_path = Path(args.input).expanduser()
    if not input_path.is_file():
        return fail("input file not found: {}".format(input_path))

    if args.output:
        out_dir = Path(args.output).expanduser()
    else:
        out_dir = input_path.parent / input_path.stem
    if out_dir.exists() and not out_dir.is_dir():
        return fail("output path exists and is not a folder: {} (use -o to choose another)"
                    .format(out_dir))
    out_html = out_dir / input_path.name
    if out_html.exists() and os.path.samefile(str(out_html), str(input_path)):
        return fail("the output would overwrite the input file; choose a different "
                    "output folder with -o")

    try:
        raw = input_path.read_bytes()
    except OSError as exc:
        return fail("cannot read {}: {}".format(input_path, exc))
    if raw.startswith(UNSUPPORTED_BOMS):
        return fail("UTF-16/UTF-32 encoded HTML is not supported; save it as UTF-8 first")

    # Latin-1 maps every byte to exactly one character, so decoding and
    # re-encoding is lossless for any ASCII-compatible encoding (UTF-8, Windows-1252,
    # Shift-JIS, ...). Only ASCII text (data URIs and the new paths) is touched.
    extractor = MediaExtractor(out_dir / MEDIA_DIR_NAME, all_types=args.all_types)
    try:
        new_raw = extractor.rewrite(raw.decode("latin-1")).encode("latin-1")
        out_dir.mkdir(parents=True, exist_ok=True)
        out_html.write_bytes(new_raw)
    except OSError as exc:
        return fail("cannot write output: {}".format(exc))

    print("Input : {} ({})".format(input_path.resolve(), format_size(len(raw))))
    print("Output: {} ({})".format(out_html.resolve(), format_size(len(new_raw))))
    if extractor.replaced:
        print("Media : replaced {} data URI(s) with {} file(s) in {} ({})".format(
            extractor.replaced, extractor.files_written, extractor.media_dir.resolve(),
            format_size(extractor.bytes_written)))
    else:
        print("Media : no embedded base64 media found; wrote an unchanged copy")
    if extractor.ignored:
        print("Note  : left {} non-media data URI(s) untouched (use --all-types to "
              "extract them too)".format(extractor.ignored))
    if extractor.skipped:
        print("Note  : {} data URI(s) could not be decoded and were left as they were"
              .format(extractor.skipped))
    return 0


if __name__ == "__main__":
    sys.exit(main())

So feel free to give it a whirl the next time you need it and let me know how it goes :D

Edited by TheDcoder

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...