Real-time News Data Stream via WebSocket Subscription (#392)

* Added News Stream * updated readme * Remove NEWS_DATA_STREAM * poetry pre-commit * update readme * test_cast * test_cast * test_cast * test_cast * test_cast * test_cast * clean up test_case.py imports * fixed news model issues * news websocket.py _dispatch fix * websocket raw_data=false testing * fixed timestamp to datetime in news ws * remove any * revert NewsClient namechange * Added TimeSeriesMixin to News model * set news.images default none * Update README.md Co-authored-by: hiohiohio <[email protected]> * resolved comments * remove test timestamp * instatiate handler per symbol * handler fix * duplicate calls * adjust code to merge * fix adjusted code --------- Co-authored-by: hiohiohio <[email protected]> Co-authored-by: Chihiro Hio <[email protected]>
alpacahq · Jul 1, 2024 · 7bdfe2f · 7bdfe2f
1 parent e092f3f
commit 7bdfe2f
Show file tree

Hide file tree

Showing 7 changed files with 197 additions and 67 deletions.
diff --git a/README.md b/README.md
@@ -6,28 +6,29 @@
 [![PyPI](https://img.shields.io/pypi/v/alpaca-py?color=blue)](https://pypi.org/project/alpaca-py/)
 
 ## Table of Contents
-* [About](#about)
-* [Documentation](#documentation)
-* [Installation](#installation)
-* [Update](#update)
-* [What's New?](#whats-new)
-   1. [Broker API](#broker-api-new)
-   2. [OOP Design](#oop-design)
-   3. [Data Validation](#data-validation)
-   4. [Many Clients](#many-clients)
-* [API Keys](#api-keys)
-   1. [Trading and Market Data API Keys](#trading-api-keys)
-   2. [Broker API Keys](#trading-api-keys)
-* [Usage](#usage)
-   1. [Broker API Example](#broker-api-example)
-   2. [Trading API Example](#trading-api-example)
-   3. [Market Data API Example](#data-api-example)
-* [Contributing](https://github.com/alpacahq/alpaca-py/blob/master/CONTRIBUTING.md)
-* [License](https://github.com/alpacahq/alpaca-py/blob/master/LICENSE)
+
+- [About](#about)
+- [Documentation](#documentation)
+- [Installation](#installation)
+- [Update](#update)
+- [What's New?](#whats-new)
+  1.  [Broker API](#broker-api-new)
+  2.  [OOP Design](#oop-design)
+  3.  [Data Validation](#data-validation)
+  4.  [Many Clients](#many-clients)
+- [API Keys](#api-keys)
+  1.  [Trading and Market Data API Keys](#trading-api-keys)
+  2.  [Broker API Keys](#trading-api-keys)
+- [Usage](#usage)
+  1.  [Broker API Example](#broker-api-example)
+  2.  [Trading API Example](#trading-api-example)
+  3.  [Market Data API Example](#data-api-example)
+- [Contributing](https://github.com/alpacahq/alpaca-py/blob/master/CONTRIBUTING.md)
+- [License](https://github.com/alpacahq/alpaca-py/blob/master/LICENSE)
 
 ## About <a name="about"></a>
 
-Alpaca-py provides an interface for interacting with the API products Alpaca offers. These API products are provided as various REST, WebSocket and SSE endpoints that allow you to do everything from streaming market data to creating your own investment apps. 
+Alpaca-py provides an interface for interacting with the API products Alpaca offers. These API products are provided as various REST, WebSocket and SSE endpoints that allow you to do everything from streaming market data to creating your own investment apps.
 
 Learn more about the API products Alpaca offers at https://alpaca.markets.
 
@@ -61,22 +62,25 @@ Run the following command in your terminal:
 ```
 
 ## What’s New? <a name="whats-new"></a>
+
 If you’ve used the previous python SDK alpaca-trade-api, there are a few key differences to be aware of.
 
 ### Broker API <a name="broker-api-new"></a>
+
 Alpaca-py lets you use Broker API to start building your investment apps! Learn more at the [Broker](https://docs.alpaca.markets/docs/about-broker-api) page.
 
 ### OOP Design <a name="oop-design"></a>
-Alpaca-py uses a more OOP approach to submitting requests compared to the previous SDK. To submit a request, you will most likely need to create a request object containing the desired request data. Generally, there is a unique request model for each method. 
 
-Some examples of request models corresponding to methods: 
+Alpaca-py uses a more OOP approach to submitting requests compared to the previous SDK. To submit a request, you will most likely need to create a request object containing the desired request data. Generally, there is a unique request model for each method.
+
+Some examples of request models corresponding to methods:
 
-* ``GetOrdersRequest`` for ``TradingClient.get_orders()``
-* ``CryptoLatestOrderbookRequest`` for ``CryptoHistoricalDataClient.get_crypto_latest_orderbook()``
+- `GetOrdersRequest` for `TradingClient.get_orders()`
+- `CryptoLatestOrderbookRequest` for `CryptoHistoricalDataClient.get_crypto_latest_orderbook()`
 
 **Request Models Usage Example**
 
-To get historical bar data for crypto, you will need to provide a ``CryptoBarsRequest`` object.
+To get historical bar data for crypto, you will need to provide a `CryptoBarsRequest` object.
 
 ```python
 from alpaca.data.historical import CryptoHistoricalDataClient
@@ -97,7 +101,8 @@ bars = client.get_crypto_bars(request_params)
 ```
 
 ### Data Validation <a name="data-validation"></a>
-Alpaca-py uses *pydantic* to validate data models at run-time. This means if you are receiving request data via JSON from a client. You can handle parsing and validation through Alpaca’s request models. All request models can be instantiated by passing in data in dictionary format.
+
+Alpaca-py uses _pydantic_ to validate data models at run-time. This means if you are receiving request data via JSON from a client. You can handle parsing and validation through Alpaca’s request models. All request models can be instantiated by passing in data in dictionary format.
 
 Here is a rough example of what is possible.
 
@@ -116,23 +121,27 @@ Here is a rough example of what is possible.
 ```
 
 ### Many Clients <a name="many-clients"></a>
-Alpaca-py has a lot of client classes. There is a client for each API and even asset class specific clients (``StockHistoricalDataClient``, ``CryptoDataStream``, ``OptionHistoricalDataClient``). This requires you to pick and choose clients based on your needs.
 
-**Broker API:** ``BrokerClient``
+Alpaca-py has a lot of client classes. There is a client for each API and even asset class specific clients (`StockHistoricalDataClient`, `CryptoDataStream`, `OptionHistoricalDataClient`). This requires you to pick and choose clients based on your needs.
+
+**Broker API:** `BrokerClient`
 
-**Trading API:** ``TradingClient``
+**Trading API:** `TradingClient`
 
-**Market Data API:**  ``StockHistoricalDataClient``, ``CryptoHistoricalDataClient``, ``OptionHistoricalDataClient``, ``CryptoDataStream``, ``StockDataStream``, ``OptionDataStream``
+**Market Data API:** `StockHistoricalDataClient`, `CryptoHistoricalDataClient`, `NewsClient`, `OptionHistoricalDataClient`, `CryptoDataStream`, `StockDataStream`, `NewsDataStream`, `OptionDataStream`
 
 ## API Keys <a name="api-keys"></a>
 
 ### Trading and Market Data API <a name="trading-api-keys"></a>
-In order to use Alpaca’s services you’ll need to sign up for an Alpaca account and retrieve your API keys. Signing up is completely free and takes only a few minutes. Sandbox environments are available to test out the API. To use the sandbox environment, you will need to provide sandbox/paper keys. API keys are passed into Alpaca-py through either ``TradingClient``, ``StockHistoricalDataClient``, ``CryptoHistoricalDataClient``, ``OptionHistoricalDataClient``. ``StockDataStream``, ``CryptoDataStream``, or ``OptionDataStream``.
+
+In order to use Alpaca’s services you’ll need to sign up for an Alpaca account and retrieve your API keys. Signing up is completely free and takes only a few minutes. Sandbox environments are available to test out the API. To use the sandbox environment, you will need to provide sandbox/paper keys. API keys are passed into Alpaca-py through either `TradingClient`, `StockHistoricalDataClient`, `CryptoHistoricalDataClient`, `NewsClient`, `OptionHistoricalDataClient`, `StockDataStream`, `CryptoDataStream`,`NewsDataStream`, or `OptionDataStream`.
 
 ### Broker API <a name="broker-api-keys"></a>
-To use the Broker API, you will need to sign up for a broker account and retrieve your Broker API keys. The API keys can be found on the dashboard once you’ve logged in. Alpaca also provides a sandbox environment to test out Broker API. To use the sandbox mode, provide your sandbox keys. Once you have your keys, you can pass them into ``BrokerClient`` to get started.
+
+To use the Broker API, you will need to sign up for a broker account and retrieve your Broker API keys. The API keys can be found on the dashboard once you’ve logged in. Alpaca also provides a sandbox environment to test out Broker API. To use the sandbox mode, provide your sandbox keys. Once you have your keys, you can pass them into `BrokerClient` to get started.
 
 ## Usage <a name="usage"></a>
+
 Alpaca’s APIs allow you to do everything from building algorithmic trading strategies to building a full brokerage experience for your own end users. Here are some things you can do with Alpaca-py.
 
 To view full descriptions and examples view the [documentation page](https://alpaca.markets/sdks/python/).
@@ -147,7 +156,7 @@ To view full descriptions and examples view the [documentation page](https://alp
 
 **Listing All Accounts**
 
-The ``BrokerClient.list_accounts`` method allows you to list all the brokerage accounts under your management. The method takes an optional parameter ``search_parameters`` which requires a ``ListAccountsRequest`` object. This parameter allows you to filter the list of accounts returned.
+The `BrokerClient.list_accounts` method allows you to list all the brokerage accounts under your management. The method takes an optional parameter `search_parameters` which requires a `ListAccountsRequest` object. This parameter allows you to filter the list of accounts returned.
 
 ```python
 from alpaca.broker.client import BrokerClient
@@ -170,8 +179,7 @@ accounts = broker_client.list_accounts(search_parameters=filter)
 
 **Submitting an Order**
 
-To create an order on Alpaca-py you must use an ``OrderRequest`` object. There are different ``OrderRequest`` objects based on the type of order you want to make. For market orders, there is ``MarketOrderRequest``, limit orders have ``LimitOrderRequest``, stop orders ``StopOrderRequest``, and trailing stop orders have ``TrailingStopOrderRequest``. Each order type have their own required parameters for a successful order.
-
+To create an order on Alpaca-py you must use an `OrderRequest` object. There are different `OrderRequest` objects based on the type of order you want to make. For market orders, there is `MarketOrderRequest`, limit orders have `LimitOrderRequest`, stop orders `StopOrderRequest`, and trailing stop orders have `TrailingStopOrderRequest`. Each order type have their own required parameters for a successful order.
 
 ```python
 from alpaca.trading.client import TradingClient
@@ -195,11 +203,11 @@ market_order = trading_client.submit_order(
                 )
 ```
 
-
 ### Market Data API Example <a name="data-api-example"></a>
+
 **Querying Historical Bar Data**
 
-You can request bar data via the HistoricalDataClients. In this example, we query daily bar data for “BTC/USD” and “ETH/USD” since July 1st 2022. You can convert the response to a multi-index pandas dataframe using the ``.df`` property.
+You can request bar data via the HistoricalDataClients. In this example, we query daily bar data for “BTC/USD” and “ETH/USD” since July 1st 2022. You can convert the response to a multi-index pandas dataframe using the `.df` property.
 
 ```python
 from alpaca.data.historical import CryptoHistoricalDataClient
@@ -223,6 +231,30 @@ bars.df
 
 ```
 
+**Querying News Data** <a name="news-client-example"></a>
+
+You can query news data via the NewsClient. In this example, we query news data for “TSLA” since July 1st 2022. You can convert the response to a pandas dataframe using the `.df` property.
+
+```python
+from alpaca.data.historical.news import NewsClient
+from alpaca.data.requests import NewsRequest
+from datetime import datetime
+
+# no keys required for news data
+client = NewsClient()
+
+request_params = NewsRequest(
+                        symbols="TSLA",
+                        start=datetime.strptime("2022-07-01", '%Y-%m-%d')
+                        )
+
+news = client.get_news(request_params)
+
+# convert to dataframe
+news.df
+
+```
+
 ### Options Trading <a name="options-trading"></a>
 
 We're excited to support options trading! Use this section to read up on Alpaca's options trading capabilities.

diff --git a/alpaca/data/historical/news.py b/alpaca/data/historical/news.py
@@ -1,14 +1,10 @@
 from typing import Optional, Union
 
-from alpaca.common.rest import RESTClient
-
 from alpaca.common.enums import BaseURL
-
-from alpaca.data.requests import NewsRequest
-
-from alpaca.data.models.news import NewsSet
-
+from alpaca.common.rest import RESTClient
 from alpaca.common.types import RawData
+from alpaca.data.models.news import NewsSet
+from alpaca.data.requests import NewsRequest
 
 
 class NewsClient(RESTClient):
@@ -62,4 +58,4 @@ def get_news(self, request_params: NewsRequest) -> Union[RawData, NewsSet]:
         if self._use_raw_data:
             return response
 
-        return NewsSet(**response)
+        return NewsSet(response)
diff --git a/alpaca/data/live/news.py b/alpaca/data/live/news.py
@@ -0,0 +1,41 @@
+from typing import Optional, Dict
+
+from alpaca.common.enums import BaseURL
+from alpaca.common.websocket import BaseStream
+
+
+class NewsDataStream(BaseStream):
+    """
+    A WebSocket client for streaming news.
+    """
+
+    def __init__(
+        self,
+        api_key: str,
+        secret_key: str,
+        raw_data: bool = False,
+        websocket_params: Optional[Dict] = None,
+        url_override: Optional[str] = None,
+    ) -> None:
+        """
+        Instantiates a WebSocket client for accessing live news.
+        Args:
+            api_key (str): Alpaca API key.
+            secret_key (str): Alpaca API secret key.
+            raw_data (bool): Whether to return wrapped data or raw API data. Defaults to False.
+            websocket_params (Optional[Dict], optional): Any parameters for configuring websocket
+                connection. Defaults to None.
+            url_override (Optional[str]): If specified allows you to override the base url the client
+                points to for proxy/testing. Defaults to None.
+        """
+        super().__init__(
+            endpoint=(
+                url_override
+                if url_override is not None
+                else BaseURL.MARKET_DATA_STREAM.value + "/v1beta1/news"
+            ),
+            api_key=api_key,
+            secret_key=secret_key,
+            raw_data=raw_data,
+            websocket_params=websocket_params,
+        )
diff --git a/alpaca/data/models/__init__.py b/alpaca/data/models/__init__.py
@@ -3,3 +3,4 @@
 from alpaca.data.models.trades import *
 from alpaca.data.models.snapshots import *
 from alpaca.data.models.orderbooks import *
+from alpaca.data.models.news import *
diff --git a/alpaca/data/models/base.py b/alpaca/data/models/base.py
@@ -24,9 +24,12 @@ def df(self) -> DataFrame:
         df = pd.DataFrame(data_list)
 
         # set multi-level index
-        # level=0 - symbol
-        # level=1 - timestamp
+        if "news" in self.dict():
+            # level=0 - id
+            df = df.set_index(["id"])
         if set(["symbol", "timestamp"]).issubset(df.columns):
+            # level=0 - symbol
+            # level=1 - timestamp
             df = df.set_index(["symbol", "timestamp"])
 
         # drop null columns
@@ -37,7 +40,7 @@ def df(self) -> DataFrame:
 
 class BaseDataSet(BaseModel):
     """
-    Base class to process data models for trades, bars and quotes.
+    Base class to process data models for trades, bars quotes, and news.
     """
 
     data: Dict[str, List[BaseModel]] = {}

diff --git a/alpaca/data/models/news.py b/alpaca/data/models/news.py
@@ -1,8 +1,12 @@
 from datetime import datetime
 from typing import Optional, List
 
+from pydantic import ConfigDict
+
 from alpaca.common.models import ValidateBaseModel as BaseModel
+from alpaca.common.types import RawData
 from alpaca.data import NewsImageSize
+from alpaca.data.models.base import BaseDataSet, TimeSeriesMixin
 
 
 class NewsImage(BaseModel):
@@ -20,43 +24,68 @@ class NewsImage(BaseModel):
 
 class News(BaseModel):
     """
-    images (URLs) related to given article
+    News article object
 
     Attributes:
         id (str): News article ID
         headline (str): Headline or title of the article
-        author (str): Original author of news article
+        source (str): Source where the news originated from (e.g. Benzinga)
+        url (Optional[str]): URL of article (if applicable)
+        summary (str): Summary text for the article (may be first sentence of content)
         created_at (datetime): Date article was created (RFC 3339)
         updated_at (datetime): Date article was updated (RFC 3339)
-        summary (str): Summary text for the article (may be first sentence of content)
+        symbols (List[str]): List of related or mentioned symbols
         content (str): Content of the news article (might contain HTML)
-        url (Optional[str]): URL of article (if applicable)
+        author (str): Original author of news article
         images (List[NewsImage]): List of images (URLs) related to given article (may be empty)
-        symbols (str): List of related or mentioned symbols
-        source (str): Source where the news originated from (e.g. Benzinga)
     """
 
-    id: float
+    id: int
     headline: str
-    author: str
+    source: str
+    url: Optional[str]
+    summary: str
     created_at: datetime
     updated_at: datetime
-    summary: str
-    content: str
-    url: Optional[str]
-    images: List[NewsImage]
     symbols: List[str]
-    source: str
+    author: str
+    content: str
+    images: Optional[List[NewsImage]] = None  # only in historical
+
+    def __init__(self, raw_data: RawData) -> None:
+        """Instantiates a news article
+
+        Args:
+            raw_data (RawData): Raw unparsed news data from API.
+        """
 
+        super().__init__(**raw_data)
 
-class NewsSet(BaseModel):
+
+class NewsSet(BaseDataSet, TimeSeriesMixin):
     """
-    images (URLs) related to given article
+    A collection of News articles.
 
     Attributes:
-        news (List[News]): Array of news objects
-        next_page_token (Optional[str]): Pagination token for next page
+        data (Dict[str, List[News]]): The collection of News articles.
     """
 
     news: List[News]
     next_page_token: Optional[str]
+
+    def __init__(self, raw_data: RawData) -> None:
+        """A collection of News articles.
+
+        Args:
+            raw_data (RawData): The collection of raw news data from API.
+        """
+        parsed_news = {}
+        articles = []
+
+        for article in raw_data.get("news", []):
+            articles.append(News(raw_data=article))
+
+        parsed_news["news"] = articles
+        parsed_news["next_page_token"] = raw_data.get("next_page_token")
+
+        super().__init__(**parsed_news)