Requirements:

• Running from source code:

• Dependent software

Capabilities:

• Download by member_id
• Download by image_id
• Download by tags
• Download from list (list.txt)
• Download from bookmarked artists (/bookmark.php?type=user)

• Download from bookmarked images (/bookmark.php)

• Download from tags list (tags.txt)
• Download new illustrations from bookmarked artist (/bookmark_new_illust.php)
• Download by Title/Caption
• Download by Tag and Member Id
• Download Member Bookmark (/bookmark.php?id=)
• Download by Group Id
• Download from supported artists (FANBOX)
• Download by artist/creator id (FANBOX)
• Download by post id (FANBOX)
• Download from followed artists (FANBOX)
• Download latest posts from supported artists (FANBOX)
• Re-encoding of all ugoira present in folder
• Batch Download from batch_job.json (experimental)

• Manage database:

• Export user bookmark (member_id) to a text files.

Docker

$ docker build -t pixivutil2 .
$ docker run -it --rm \
  -v $(pwd):/workdir \
  -w /workdir \
  pixivutil2 \
  /bin/bash -c "python PixivUtil2.py"

WARNING

FAQs

A. Usage

Q1. How to paste Japanese tags to the console window?
    - Click the top-left icon -> select Edit -> Paste (Cannot use Ctrl-V), if
      it show up as question mark -> Change the Language for non-Unicode
      program to Japanese (google it).
    - or use online url encoder (http://meyerweb.com/eric/tools/dencoder/)
      and paste the encoded tag back to the console.
    - or paste it to tags.txt and select download by tags list. Separate each
      tags with space, and separate with new line for new query.
Q2. My password doesn't show up in the console!
    - This is normal. The program still reads it.
    - or you can put in the config.ini if not sure.
Q3. I cannot login to Pixiv!
    - Check your password.
    - Try to login to the Pixiv Website.
    - Try to use the config.ini on the [Authentication] section.
    - Check your date and time setting (e.g.: https://www.timeanddate.com/)
    - Disable Daylight Saving Time and try again.
    - Copy your session values from browser:
      1. Open Firefox.
      2. Go to Pixiv website and login, remember to enable [Remember Me]
          check box.
      3. Press F12 to open Developer Tools, and select the Storage tab.
      4. Click the Cookies and select for the pixiv.net.
      5. Look for Cookie named = PHPSESSID.
      6. Copy the content value. https://imgur.com/a/BppHOoQ
      7. Open config.ini, go to [Authentication] section, paste the value
         to cookie. https://imgur.com/VB2g3qn
Q4. PixivUtil working from local terminal on Linux box but not working when I
    used SSH with PuTTY!
    - export LANG=en_US.UTF-8. PuTTY does not set locales right, when they are
      not set, python does not know what to write (Thanks to nho!)
    - ... and export PYTHONIOENCODING=utf-8, so it can create DB and populate
      it properly (Thanks to Mailia!)
Q5. How to delete member id from Database?
    - Open the application and choose Manage Database (d) then select delete
      Member by Member Id.
    - Open the database (db.sqlite) directly using sqlite browser and use sql
      command to delete it.
    - If you are downloading using Download from List.txt (3), you can create
      ignore_list.txt to skip the member id.
Q6. The app doesn't download all the images! (I want to download SFW images too).
    - Pixiv only allow to search up to 1000 pages if you don't have Pixiv
      Premium.
    - Check your pixiv website settings (refer to https://goo.gl/gQi09v),
      then delete the cookie value in config.ini and retry.
    - Check the value of r18mode in config.ini. Setting it to True will only
      download R-18 images.
Q7. The apps show square/question mark texts in the console output!
    - This is because your Windows is not set to Japanese for the Regional Settings
      in control panel.
    - Since 20161114+ version, you need to set the console font properties to
      use font with unicode support (e.g. Arial Unicode, MS Gothic).
Q8. Where to get FFmpeg software? How to enable createwebm?
    - Download the stable version of FFmpeg from https://www.ffmpeg.org/download.html.
    - For Windows:
      - Extract the archive to a folder.
      - Open the extracted folder and open to the /bin folder.
      - Copy the application ffmpeg.exe to your PixivUtil2 folder.
    - For Linux:
      - Install the package using your favorite package manager.
Q9. The downloaded images are corrupted, how to redownload it again?
    - You can delete the download history in databases by manually delete the image id
      from databases (enter d, followed by 10).
    - Or, you can set alwaysCheckFileSize = True and verifyimage = True in config.ini
      and retry the download.
      
Q10. I got this error またはメールアドレス、パスワードが正しいかチェックしてください。
    - Use your email address for the username, or check your password in config.ini
Q11. Older windows support (e.g. Win7)?
    - You can try to run from source code with the latest supported python 3.x.
      See the instruction here: https://github.com/Nandaka/PixivUtil2/wiki/IDE-Enviroment-(Windows)
      
Q12. How do I get around Cloudflare preventing me from downloading from Fanbox
    - Go to https://www.whatismybrowser.com/detect/what-is-my-user-agent/ and copy your user agent to the useragent field in config.ini
    - Go to https://curl-cffi.readthedocs.io/en/latest/impersonate/targets.html to see a list of supported browsers for impersonation. Inside config.ini, replace userAgentImpersonation with the appropriate browser you would like to impersonate. If you do not know which browser to use, just use the latest one that closely matches your user agent.
    - Change your user agent's version in config.ini to match the browser you are impersonating. For example, Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:150.0) Gecko/20100101 Firefox/150.0 => Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:150.0) Gecko/20100101 Firefox/147.0 if you are impersonating firefox147
    - Have your browser's network tab open and visit fanbox.
    - Filter your network requests by "post" and click on any network request
    - Go to the "headers" tab on the ribbon and scroll down to the "request headers" section.
    - Enable "raw" text to get the real string values
    - Copy the value from "Cookie" into cookieFanboxTemp in config.ini. Your cookie should be a long string that looks like p_ab_id=<omitted>; p_ab_id_2=<omitted>; p_ab_d_id=<omitted>; cf_clearance=<omitted>; privacy_policy_agreement=<omitted>; privacy_policy_notification=<omitted>; __cf_bm=<omitted>; FANBOXSESSID=<omitted> ...
    - You may have more or fewer fields for your cookie. This is normal.
    - If these steps stopped working, you probably:
      - Logged out on fanbox
      - Changed your IP
      - Are using an IP address that is specifically blacklisted by Fanbox
      - Are on the Japanese site for fanbox

B.Bugs/Source Code/Supports

Q1. Where I can report bugs?
    - Please report any bug to https://github.com/Nandaka/PixivUtil2/issues.

Q2. Where I can support/donate to you?
    - You can send it to my PayPal account (nchek2000[at]gmail[dot]com).
    - or visit https://bit.ly/PixivUtilDonation.
Q3. I want to use/modify the source code!
    - Feel free to use/modify the source code as long you give credit to me
      and make the modificated source code open.
    - if you want to add feature/bug fix, you can do fork the repository in
      https://github.com/Nandaka/PixivUtil2 and issue Pull Requests.
Q4. I got ValueError: invalid literal for int() with base 10: '<something>'
    - Please modify _html.py from mechanize library, search for
      'def unescape_charref(data, encoding):' and replace with patch in
      https://pastebin.com/5bT5HFkb.
Q5. I got '<library_name> module no found error'
    - Download the library from the source (see links from the Requirements
      section) and copy the file into your Lib\site-packages directory.
    - Or use pip install (google on how to use).

C.Log Messages

Q1: HTTPError: HTTP Error 404: Not Found
    - This is because the file doesn't exist in the pixiv server, usually
       because there is no big images version for the manga mode (currently the
       apps will try to download the big version first then try the normal size
       if failed, this is only for the manga mode and it is normal).

Q2: Error at process_image(): (<type 'exceptions.WindowsError'>, WindowsError
    (32, 'Prosessi ei voi kayttaa tiedostoa, koska se on toisen prosessin
    kaytossa')
    - The file is being used by another process (google translate). Either you
      ran multiple instace of Pixiv downloader from the same folder, or there
      are other processes locking the file/db.sqllite (usually from antivirus
      or some sync/backup application).
Q3: Error at process_image(): (<type 'exceptions.AttributeError'>,
    AttributeError ("'NoneType' object has no attribute 'find'",)
    - Usually this is because of failed login (cookie not valid). Try to change
      your password to simple one for testing, or copy the cookie from browser:
      1. Open Firefox/Chrome.
      2. Login to your Pixiv.
      3. On Pixiv page, press F12 and choose the Storage tab (Firefox), or
         Right click on the leftmost address bar/the (i) icon (Chrome)
      5. Click the View Cookies button.
      6. Look for Cookie named = PHPSESSID.
      7. Copy the content value.
      8. Open config.ini, go to [Authentication] section, paste the value to
         cookie.
    - Or because Pixiv has changed the layout code, so the Pixiv
      downloader cannot parse the page correctly. Please tell me by posting a
      comment if this happens and include the details, such as the member/image
      id, dump html, and log file (check on the application folder).
Q4: URLError: <urlopen error [Errno 11004] getaddrinfo failed>
    - Update version to > pixivutil20221029.
    - This is because the Pixiv downloader cannot resolve the address to
      download the images, please try to restart the network connection or do
      ipconfig /flushdns to refresh the dns cache (windows).
Q5: Error at download_image(): (<class 'socket.timeout'>, timeout('timed out',)
    - This is because the Pixiv downloader didn't receive any reply for
      specified time in config.ini from Pixiv. Please retry the download again
      later.
Q6: httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
    - Set userobots = False in config.ini

Command Line Option

-h, --help            show this help message and exit
  -s STARTACTION, --startaction=STARTACTION
                        Action you want to load your program with:
                        1 - Download by member_id
                            (required: list of member_ids separated by space
                             optional: --include_sketch to also download Pixiv Sketch)
                        2 - Download by image_id
                            (required: followed by image_ids separated by space)
                        3 - Download by tags
                            (required: tags
                             optional: --use_wildcard_tag, --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        4 - Download from list
                            (required: -f LIST_FILE and followed with optional tag)
                        5 - Download from user bookmark
                            (optional: -p BOOKMARK_FLAG [y/n/o] for private bookmark, --sp=START_PAGE, and --ep=END_PAGE)
                        6 - Download from image bookmark
                            (required: -p BOOKMARK_FLAG [y/n/o] for private bookmark
                             optional: --sp=START_PAGE, and --ep=END_PAGE, and followed with tag)
                        7 - Download from tags list
                            (required: -f LIST_FILE,
                             optional: --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        8 - Download new illust from bookmark
                            (optional: --sp=START_PAGE, and --ep=END_PAGE)
                        9 - Download by Title/Caption
                            (required: title/caption
                             optional: --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        10 - Download by Tag and Member Id
                            (required: member_id, followed by tags
                             optional: --sp=START_PAGE, and --ep=END_PAGE)
                        11 - Download Member's Bookmarked Images
                            (required: followed by member_ids separated by space)
                        12 - Download by Group ID
                            (required: Group ID, limit, and process external[y/n])
                        13 - Download by Manga Series ID
                            (required: Manga Series ID separated by space
                            optional: --sp=START_PAGE, and --ep=END_PAGE)
                        f1 - Download from supported artists (FANBOX)
                            (optional: End Page)
                        f2 - Download by artist/creator id (FANBOX)
                            (required: artist(digits only)/creator ids separated by space,
                             optional: end page)
                        f3 - Download by post id (FANBOX)
                            (required: post ids, separated with space)
                        f4 - Download from followed artists (FANBOX)
                            (optional: End Page)
                        f5 - Download from custom artist list (FANBOX)
                            (optional: End page, path to list)
                        f7 - Download latest posts from supported artists (FANBOX)
                            (optional: number of pages greater than 0, default: 1)
                        b - Batch Download from batch_job.json (experimental)
                            (optional: --bf=BATCH_FILE)
                        l - Export local database image_id/post_id
                            (required: --up=USE_PIXIV, and --uf=USE_FANBOX, and --us=USE_SKETCH)
                        e - Export online bookmark
                            (required: -p BOOKMARK_FLAG [y/n/o] for private bookmark,
                             optional: --ef=EXPORT_FILENAME)
                        m - Export online user bookmark
                            (required: member_id, optional: --ef=EXPORT_FILENAME)
                        d - Manage database
  -x, --exitwhendone    Exit programm when done.
                        (only useful when DB-Manager)
  -i, --irfanview       start IrfanView after downloading images using
                        downloaded_on_%date%.txt
  -n NUMBEROFPAGES, --numberofpages=NUMBEROFPAGES
                        temporarily overwrites numberOfPage set in config.ini
  -c [PATH], --config [PATH] provide different config.ini

Error Codes

• 100 = Not Logged in.
• 1001 = User ID not exist/deleted.
• 1002 = User Account is Suspended.
• 1003 = Unknown Member Error.
• 1004 = No image found.
• 1005 = Cannot login.
• 2001 = Unknown Error in Image Page.
• 2002 = Not in MyPick List, Need Permission.
• 2003 = Public works can not be viewed by the appropriate level.
• 2004 = Image not found/already deleted.
• 2005 = Image is disabled for under 18, check your setting page (R-18/R-18G).
• 2006 = Unknown Image Error.
• 9000 = Download Failed.
• 9001 = Download Failed: Harddisk related.
• 9002 = Download Failed: Network related.
• 9005 = Server Error.

config.ini

[Authentication]

• username

• password

• cookie

• cookieFanbox

• cookieFanboxTemp

• refresh_token

• userAgentImpersonation

[Pixiv]

• numberofpage

• r18mode

• r18Type

• dateformat

• autoAddMember

• autoAddTag

• autoAddCaption

• autoAddStats

• aiDisplayFewer

[FANBOX]

• filenameFormatFanboxContent

• filenameFormatFanboxCover

• filenameFormatFanboxInfo

• writeHtml

• minTextLengthForNonArticle

• minImageCountForNonArticle

• useAbsolutePathsInHtml

• downloadCoverWhenRestricted

• checkDBProcessHistory

• checkUpdatedLimitFanbox

• listPathFanbox

[Network]

• useproxy

• proxyaddress

• useragent

• userobots

• timeout

• retry

• retrywait

• downloadDelay

• checkNewVersion

• notifyBetaVersion

• openNewVersion

• enableSSLVerification

[Debug]

• logLevel

Two log files are written next to the application: pixivutil.log with everything at the configured level, and pixivutil_error.log with only warnings and errors, so a failed run can be reviewed without searching the full log. Both rotate at 10MB, keeping 10 backups, and both are suppressed by disableLog.

• enableDump

• skipDumpFilter

• dumpMediumPage

• dumpTagSearchPage

• debughttp

[IrfanView]

• IrfanViewPath

• startIrfanView

• startIrfanSlide

• createDownloadLists

[Settings]

• downloadlistdirectory

• uselist

• processfromdb

• rootdirectory

• downloadavatar

• usesuppresstags

• tagsLimit

• writeImageJSON

• writeimageinfo

• writeRawJSON

• RawJSONFilter

• includeSeriesJSON

• writeImageXMP

• writeImageXMPPerImage

.gif.xmp

.webp.xmp

• verifyimage

• writeUrlInDescription

-> %member_token%
   Member token, might change.
-> %member_id%
   Member id, in number.
-> %artist%
   Artist name, might change too.
-> %urlFilename%
   The actual filename stored in server without the file extensions.
-> %date%
   Current date in YYYYMMMDD format.
-> %date_fmt{format}%
   Current date using custom format.
   Use Python string format notation, refer: https://goo.gl/3UiMAb
   e.g. %date_fmt{%Y-%m-%d}%
-> %image_ext%
   The image's file extension (jpg, png, etc.), the "." is not included.
   The correct file extension is already appended to the end of all files.
   This is available if you want to add more, or want to add the image's file extension to info files etc.

-> %image_id%
   Image id, in number. (Post id for FANBOX and sketches)
-> %title%
   Image title, usually in japanese character.
-> %tags%
   Image tags, usually in japanese character. (not implemented for FANBOX yet)
-> %works_date%
   Works date, complete with time.
-> %works_date_only%
   Only the works date.
-> %works_date_fmt{<format>}%
   works date using custom format.
   Use Python string format notation, refer: https://goo.gl/3UiMAb
   e.g. %works_date_fmt{%Y-%m-%d}%
-> %works_res%
   Image resolution, will be containing the page count if manga.
-> %works_tools%
   Tools used for the image.
-> %R-18%
   Append R-18/R-18 based on image tag, can be used for creating directory
   by appending directory separator, e.g.: %R-18%\%image_id%.
-> %page_big%
   for manga mode, add big in the filename.
-> %page_index%
   for manga mode, add page number with 0-index. It will auto-pad with 0 based on the total count.
-> %page_number%
   for manga mode, add page number with 1-index. It will auto-pad with 0 based on the total count.
-> %bookmark%
   for bookmark mode, add 'Bookmarks' string.
-> %original_member_id%
   for bookmark mode, put original member id.
-> %original_member_token%
   for bookmark mode, put original member token.
-> %original_artist%
   for bookmark mode, put original artist name.
-> %searchTags%
   for download by tags and bookmarked images, put searched tags.
-> %bookmark_count%
   Bookmark count, will have overhead except on download by tags.
-> %image_response_count%
   Image respose count, will have overhead except on download by tags.
-> %manga_series_order%
   the order in the manga series.
-> %manga_series_id%
   original manga series id.
-> %manga_series_title%
   original manga series title, different from work title.
-> %AI%
   Add 'AI' for AI-generated images (aiType==2).

-> %sketch_member_id%
   Pixiv Sketch artist id, might be different from Pixiv's artist id.

-> %fanbox_name%
   Fanbox name, might be different from Pixiv's artist name.
   Useful if the artist is suspended from Pixiv and there is no record in the DB to avoid interuption.

member_id1 directory1
member_id2 directory2
  ...
#comment - lines starting with # will be ignored

### START EXAMPLE LIST####
this is a comment line, lines starting with # will be ignored
here is the first member:
123456
you can see, the line has only the member id
usually I use it the following way:
#
username (so I can recognize it ;) )
123456
#
next 2 lines contain a special folder for this member
123456 .\test
123456 ".\test"
now all images from member no. 123456 will be safed in directory "test" in the
same directory as PixivUtil2
as you can see you can use it with "" or without ;)
#
next will be stored at the same partition as PixivUtil, but the directory is
located in root-part of it
123456 \test
123456 "\test"
this will lead to "C:\test" when pixivUtil is located on "C:\"
#
next line uses complete path to store the files
123456 F:\new Folder\test
123456 "F:\new Folder\test"
this will set the folder everywhere on your partitions
#
123456 %root%\special folder
123456 "%root%\special folder"
this will set the download location to "special folder" in your rootDirectory
given in config
http://www.pixiv.net/member.php?id=123456
http://www.pixiv.net/member_illust.php?id=123456
also support url format.
END EXAMPLE LIST####

-> %coverImage%
   A 'div' tag with its 'class' set to 'cover', and a child 'img' tag with 
   the url to the cover image as its 'src' attribute.
-> %coverImageUrl%
   Simply the url to the cover image in clear text.
-> %artistName%
   Same as %artist% in 'Filename Format Syntax' in clear text.
-> %imageTitle%"
   Title of the post in clear text.
-> "%worksDate%"
   Published date of the post in clear text.
-> %body_text(article)%
   This works for article type posts only.
   A 'div' tag with its 'class' set to 'article', and the post's content,
   which is already formatted HTML if the post is article, as its inner text.
-> %images(non-article)%
   This works for none-article type posts only.
   A 'div' tag with its 'class' set to 'non-article images', and 'a' tags
   of all files in the post as its children tokens.
   For each 'a' tag, its 'href' would be url to the file, and the inner text
   would be an 'img' tag with its 'src' set to the url to the file if the
   file's extension is 'jpg', 'jpeg', 'png' or 'bmp'. Otherwise the inner text
   would simply be the url to the file.
-> %text(non-article)%
   This works for none-article type posts only.
   A 'div' tag with its 'class' set to 'non-article text' and all paragraphs
   of text put in 'p' tags as its children tokens.

-> %replace<default>(your_default_replace_with)%
   Use this syntax to define default value to replace with.
   If this syntax gets used multiple times in the option value, the first value would be used.
   If this value is not set, "_" would be used.
-> %pattern<you_group_name>(your_pattern)%
-> %replace<you_group_name>(your_replace_with)%
   Use these two syntaxes to set groups of rules. Supports regular expression.
   You should not use "default" as group names, otherwise the first replace would
   be parsed as default value to replace with, while the others would be ignored.
   Groups with no "pattern" would be ignored.
   Groups with no "replace" use default value.
   If multiple "pattern"s or "replace"s share the same group name, the last value set
   would be used.

# If you just want to replace some single chars with "_"
\@[]
If you want to replace them with "@":
\@[]%replace<default>(@)%
If you want to replace certain words:
This example would first replace all "maze" with "labyrinth",
then all "labyrinth" with "nevermind"
%pattern<1>(maze)%%replace<1>(labyrinth)%%pattern<2>(labyrinth)%%replace<2>(nevermind)%
If you want to replace characters within certain unicode range,
then remove all continuous "_"s with a single "_":
%pattern<unicode>([\U0001d400-\U0001ffff])%%pattern<1>(_+)%%replace<1>(_)%

pip install --user pytest
pytest -v ./test_*