Check the Content-Type on the final GET response before choosing an extraction parser. Keep the original header, normalize its type/subtype, and route only the media types your application explicitly supports. For a small reader, that may mean separate HTML, JSON, PDF, and plain-text paths, with an unresolved result for missing or ambiguous declarations. Then validate the selected parser's output and the document you intended to retrieve. A successful HTTP response, a .html suffix, or readable-looking bytes cannot complete those checks for you.

Inspect the response you will actually consume

A source URL can lead through a redirect to a different resource. Store the requested URL and final URL alongside the status and headers of the response whose bytes you parse. If a request fails, retain that failure before deciding whether its error body is useful for diagnosis.

RFC 9110 defines Content-Type as representation metadata. It describes the declared media type; servers can omit or misconfigure it. Your application therefore needs an explicit policy for uncertain declarations. Silently treating every response as HTML makes a later extraction failure harder to explain.

For a permitted public page, curl can save the headers and body separately:

curl --silent --show-error --location \
  --dump-header response.headers --output response.body \
  "https://anycrawler.com/crawler/page/fetch/"

With redirects, the header file can contain several response blocks. Read the final block for the saved body, and retain earlier blocks when you need a redirect audit. This command follows redirects; a production service accepting user-supplied URLs also needs its own destination and redirect policy.

HEAD can reduce body transfer during a preliminary check, but it has no response body and some dynamically determined fields may be omitted. Check the GET you actually consume as well. A separate request can select a different response; its metadata should stay attached to that response.

Choose a parser without discarding the declaration

The type/subtype is the routing input in this example. In text/html; charset=utf-8, keep the complete field while comparing the normalized text/html part. Parameters can matter to later processing. MDN's media-type guide explains the syntax and why a URL's filename extension does not determine the returned representation.

The table below combines a deliberately small routing policy with observations from individual public requests. The headers are snapshots of those responses, not promises about every future request.

Observed declaration and example Candidate path What must still pass
text/html — AnyCrawler Fetch documentation HTML extraction Decode and parse the body; confirm the intended page and required text
application/json; charset=utf-8 — public CMS categories response JSON parser Decode valid JSON; check the expected object and fields
application/pdf; qs=0.001 — W3C dummy PDF test resource PDF-aware pipeline Validate the document and extraction output; preserve page context when needed
text/plain — AnyCrawler robots file Plain-text processing Decode the text and check the expected content
Missing, duplicate, malformed or unsupported declaration Unresolved Record why routing stopped; use a reviewed exception only when justified

An HTML parser can successfully parse a login page or an access notice. That success alone would not establish that the requested article was retrieved. A JSON parser can likewise return a valid error object. Parser success and task success need separate fields in your result.

A small Python routing function

Pass your HTTP client's status and all Content-Type field values to this function. It checks a media-type essence and returns a proposed parser. It does not inspect the body or validate the full parameter grammar.

import re

TOKEN = r"[!#$%&'*+.^_`|~0-9A-Za-z-]+"
PARSERS = {
    "text/html": "html",
    "application/json": "json",
    "application/pdf": "pdf",
    "text/plain": "text",
}

def choose_parser(status, content_types):
    if status != 200:
        return {"parser": None, "reason": "http_status"}
    if len(content_types) != 1:
        return {"parser": None, "reason": "missing_or_duplicate_type"}
    raw = content_types[0]
    media_type = raw.split(";", 1)[0].strip().lower()
    if "," in raw or not re.fullmatch(TOKEN + "/" + TOKEN, media_type):
        return {"parser": None, "reason": "ambiguous_or_invalid_type"}
    parser = PARSERS.get(media_type)
    return {
        "media_type": media_type,
        "parser": parser,
        "reason": "declared_type" if parser else "unsupported_type",
    }

print(choose_parser(200, ["Text/HTML; charset=UTF-8"]))
print(choose_parser(200, []))
print(choose_parser(403, ["text/html"]))

The first call selects html; the others retain an unresolved parser with a reason. The same function was run against the captured response headers in the table and synthetic cases for missing, duplicate, combined, malformed and unsupported types, plus non-200 statuses. Those cases check the routing policy, not the success rate of a crawler or document parser.

Several choices are intentionally conservative:

  • The sample routes only status 200. Partial and no-content statuses need separate handling; transport completeness still needs checking.
  • Any comma triggers deferral. That catches a combined type list, but also defers a valid quoted parameter containing a comma. A more capable field parser may support that case explicitly.
  • application/octet-stream, vendor types and additional formats remain unsupported until you add a reviewed handler. No fallback parser is selected from the filename.

Keep the original field values outside the function's return value. Also retain the final URL, retrieval time, status, content encoding, selected parser, parser error and whether required content was found. If you save a body hash, specify which bytes you hashed. Compressed transfer bytes, decoded bytes and normalized Markdown are different artifacts.

The function does not decompress or decode anything. Before parsing, your HTTP client and format handler must correctly handle the received content encoding and the format's character-encoding rules. A candidate parser label should never be used as permission to decode arbitrary bytes as UTF-8.

Test the HTML path through AnyCrawler

Once a response belongs on an HTML path, check whether extraction yields the text your task needs. The free URL-to-Markdown preview lets you try a public page without an API key. This request was run for the same permitted documentation page used in the header example:

curl --silent --show-error --get \
  --data-urlencode "url=https://anycrawler.com/crawler/page/fetch/" \
  "https://api.anycrawler.com/free/v1/crawl"

The recorded response returned the expected requested URL and page title, status_code: 200, and Markdown containing “Convert a URL into clean Markdown”. That is a useful content check for this example. It does not establish how another page, a PDF or a JavaScript application will behave.

Notice the two representations involved: the preview endpoint returns a JSON envelope, while results.markdown contains text extracted from the target page. The envelope's application/json header describes the API response. It cannot be used as the target page's original Content-Type.

The free response in this sample did not include final_url or credits_used; an evidence record should leave those values absent or null. Its fields also do not expose the target's raw response headers. Preserve upstream-header observations separately when your workflow requires them. The preview can return cached results, so its timestamp and an independent direct request should not be treated as the same capture.

For a production integration where the required text is already in returned HTML, use the explicit Fetch extraction contract. If that text appears only after JavaScript execution, evaluate Render and then validate the result again. Media-type routing does not decide whether JavaScript is necessary. Neither path should be presented as a PDF parser, a login interaction workflow or a way to bypass access controls. Search finds candidate URLs; Screenshot records visible state; browser automation handles supported interactions. Those are separate jobs.

When should an unresolved response become a supported exception?

A strict allowlist will defer some useful sources. A source-specific adapter can be reasonable when you control the input set and can explain a stable mismatch between the declaration and the actual document. The question is what evidence you will require before enabling that adapter, and how you will notice when the source changes.

Keep such an exception narrow: retain the original declaration, the expected source identity, the parser chosen by the adapter and the checks that accepted its output. Decide who reviews a changed response and which failures return it to unresolved. Each additional format needs a tested processing path, with a separate unresolved outcome for anything still unsupported.

Frequently asked questions

Should I use HEAD or GET to check Content-Type?

Use GET metadata when deciding how to process the body you actually retrieved. HEAD is useful as a preliminary check when you want metadata without transferring the body, but it cannot validate content. Keep the methods separate in your evidence. If HEAD omits a field or a later GET differs, inspect that GET's status and headers before selecting its processing path.

Can I identify HTML or PDF from the URL extension?

A suffix can help a person recognize a candidate source, but the response still needs inspection. A URL ending in .pdf may return an access page or an error, and an extensionless URL may serve a document. Store the requested URL as provenance, then use the actual response declaration and format-specific checks to decide what your application can process.

What should happen when Content-Type is missing?

Keep the declaration missing in the record and return an unresolved routing state. Decide separately whether a narrowly defined adapter or a format-detection step is appropriate for that source. Do not silently fill in HTML just because that is your usual input. Preserving the uncertainty lets later reviewers distinguish what the server declared from what your application inferred and why.

Does a JSON extraction API response mean the source page was JSON?

No. A JSON envelope can carry Markdown extracted from an HTML page, an error message, metadata or other results. Its HTTP Content-Type applies to that envelope. Inspect documented fields for information about the target; leave unavailable target metadata unknown. In the tested free preview, the returned Markdown supported the content check, while raw target headers and some provenance fields were unavailable.

Does text/html mean I should use browser rendering?

The media type alone does not answer that question. Inspect whether the returned HTML contains the text your task needs. If it does, an HTML-first extraction path may be enough. If the required content appears only after JavaScript runs, evaluate rendering. Then check the intended page and required fields again; switching to a browser does not remove the need for content validation.

Can Content-Type checks prove that a document is safe or accurate?

They can help select a processing path, but they do not establish safety, authorship or factual accuracy. The declaration can be wrong, and a successfully parsed document can still contain irrelevant or misleading text. Keep parser controls and source review appropriate to your application. Before giving extracted content to an agent, preserve its status as external evidence and independently control the actions that agent can take.