Scraper
=======

``richy.core.scraper`` contains the classes responsible for actual
web content scraping. It wraps the
`Rug <https://gitlab.com/imn1/rug>`_ library (stock-market data) and
`Karpet <https://gitlab.com/imn1/karpet>`_ (crypto data), plus
`yfinance <https://github.com/ranaroussi/yfinance>`_ as a fallback
where applicable.

CurrentPrice
------------

Frozen dataclass returned by
:meth:`~richy.core.scraper.Manager.get_current_price_and_change`. Holds
the current price plus the day's change in both absolute and
percentage form, the market state (``open`` / ``closed`` /
``pre-market`` / ``post-market``), and -- when the market is in
pre/post-market -- a ``closed_market`` snapshot of the most recent
regular-session close.

.. autoclass:: richy.core.scraper.CurrentPrice
   :show-inheritance:

.. automethod:: richy.core.scraper.CurrentPrice.is_closed
.. automethod:: richy.core.scraper.CurrentPrice.round

Manager
-------

Central scraping facade. Every periodic task in
:doc:`/architecture/tasks` ultimately delegates to one of the methods
below. Methods are static -- ``Manager`` carries no per-instance
state.

.. autoclass:: richy.core.scraper.Manager
   :show-inheritance:

Basic info
~~~~~~~~~~

Per-item *basic info* lookups (name, market, market cap, P/E, EPS,
year low / high, holdings count, etc.). Results are written into
:class:`~richy.core.models.ItemData` by the relevant celery task.

.. automethod:: richy.core.scraper.Manager.get_share_basic_info
.. automethod:: richy.core.scraper.Manager.get_etf_basic_info
.. automethod:: richy.core.scraper.Manager.get_coin_basic_info

Current price
~~~~~~~~~~~~~

.. automethod:: richy.core.scraper.Manager.get_current_price_and_change

Price history
~~~~~~~~~~~~~

Bulk price fetchers. Each returns a ``pandas.DataFrame`` ready to
upsert into :class:`~richy.core.models.Price`. ``fetch_intraday_prices``
returns the past-day, fine-grained data used by the *Eye* component.

.. automethod:: richy.core.scraper.Manager.fetch_share_prices
.. automethod:: richy.core.scraper.Manager.fetch_etf_prices
.. automethod:: richy.core.scraper.Manager.fetch_index_prices
.. automethod:: richy.core.scraper.Manager.fetch_coin_prices
.. automethod:: richy.core.scraper.Manager.fetch_intraday_prices

Financials and ratings
~~~~~~~~~~~~~~~~~~~~~~

Per-share financial data feeds. Each call upserts a row into
:class:`~richy.core.models.Asset` with the appropriate ``type``.

.. automethod:: richy.core.scraper.Manager.fetch_financials
.. automethod:: richy.core.scraper.Manager.fetch_ratings
.. automethod:: richy.core.scraper.Manager.fetch_price_ratings

Dividends
~~~~~~~~~

.. automethod:: richy.core.scraper.Manager.get_dividends

Holdings
~~~~~~~~

.. automethod:: richy.core.scraper.Manager.fetch_etf_holdings

Downloader
----------

General-purpose HTTP client builder. ``Downloader.get_client()``
returns a pre-configured ``requests.Session`` for code that fetches
URLs directly (outside of what Rug / Karpet / yfinance already wrap).
The session ships with:

* Automatic retries on connection errors and configurable HTTP status
  codes (defaults: ``429``, ``500``, ``502``, ``503``, ``504``).
* Exponential backoff between attempts, ``Retry-After`` honored.
* A per-request default timeout enforced through a custom HTTP
  adapter (:class:`~richy.core.scraper.Downloader.TimeoutHTTPAdapter`)
  so callers that forget ``timeout=`` cannot hang indefinitely.
* A desktop browser ``User-Agent`` header to avoid naive bot filters.

Module-level constant ``DEFAULT_TIMEOUT`` (= 10 seconds) is the
fallback timeout used when no explicit value is passed.

.. note::

   ``urllib3``'s default ``allowed_methods`` is preserved, so retries
   apply only to idempotent verbs (``HEAD``, ``GET``, ``PUT``,
   ``DELETE``, ``OPTIONS``, ``TRACE``). ``POST`` and ``PATCH`` are
   **not** retried.

.. autoclass:: richy.core.scraper.Downloader
   :show-inheritance:

.. autoclass:: richy.core.scraper.Downloader.TimeoutHTTPAdapter
   :show-inheritance:

.. automethod:: richy.core.scraper.Downloader.get_client